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Abstract :  As  a  consequence  of  the  reluctance  by  DoD  to  conduct  realistic  live  fire 
tests  on  systems  in  development,  Congress  passed  legislation  in  fiscal  year  1987 
that  requires  realistic  vulnerability  testing  of  covered  weapon  systems,  including 
aircraft,  before  they  can  proceed  beyond  Low  Rate  Initial  Production.  The 
legislation  is  known  as  the  Live  Fire  Test  (LFT)  law.  The  realistic  vulnerability 
tests  mandated  by  the  LFT  law  must  be  conducted  on  the  system,  configured  for 
combat,  using  weapons  likely  to  be  encountered  in  combat.  A  waiver  from  these 
mandated  tests  is  possible  if  the  Secretary  of  Defense  certifies  to  Congress,  prior 
to  entry  into  engineering  and  manufacturing  development,  that  such  tests  are 
unreasonably  expensive  and  impractical.  An  alternate  plan  for  assessing 
vulnerability  must  be  submitted  if  a  waiver  to  the  mandated  system  tests  is  given. 
In  an  attempt  to  resolve  the  controversy  regarding  the  law’s  requirements,  and  to 
obtain  an  independent  opinion  regarding  the  total  vulnerability  assessment  process 
and  the  law's  contribution  to  that  process,  the  DDDR&E (T&E) ,  now  the  Director,  Test 
and  Evaluation,  requested  that  the  National  Research  Council  (NRC)  conduct  a  study 
of  aircraft  vulnerability  assessment.  The  NRC  appointed  the  "Committee  on  Weapons 
Effects  on  Airborne  Systems  to  conduct  the  study.  The  report  is  the  result  of  the 
committee's  deliberations. 
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Preface 


The  combat  vulnerability  of  a  military  vehicle,  that  is,  the  inability  of  the  vehicle  to 
withstand  the  damage  caused  by  a  man-made  hostile  environment,  is  a  critical  system 
characteristic.  A  vehicle’s  vulnerability  can  be  assessed  or  evaluated  by  the  use  of  analytical 
models  and  by  empirical  live  fire  testing.  Many  within  the  Department  of  Defense  (DoD) 
hold  the  opinion  that  live  fire  testing  can  be  very  expensive  relative  to  the  value  of  the  results 
obtained.  Consequently,  there  has  been  a  reluctance  within  DoD  to  conduct  realistic  tests 
on  new  systems  during  their  development.  This  reluctance  was  brought  into  the  public  domain 
in  the  mid-1980s  when  the  U.S.  Army  was  accused  of  not  conducting  realistic  vulnerability 
tests  on  its  Bradley  Fighting  Vehicle. 

As  a  consequence  of  this  reluctance  by  DoD  to  conduct  realistic  live  fire  tests  on  systems 
in  development.  Congress  passed  legislation  in  fiscal  year  1987  that  requires  realistic 
vulnerability  testing  of  covered  weapon  systems,  including  aircraft,  before  they  can  proceed 
beyond  Low  Rate  Initial  Production.  This  legislation  is  known  as  the  Live  Fire  Test  (LFT) 
law.  The  realistic  vulnerability  tests  mandated  by  the  LFT  law  must  be  conducted  on  the 
system,  configured  for  combat,  using  weapons  likely  to  be  encountered  in  combat.  A  waiver 
from  these  mandated  tests  is  possible  if  the  Secretary  of  Defense  certifies  to  Congress,  prior 
to  entry  into  engineering  and  manufacturing  development,  that  such  tests  are  unreasonably 
expensive  and  impractical.  An  alternate  plan  for  assessing  vulnerability  must  be  submined 
if  a  waiver  to  the  mandated  system  tests  is  given. 

In  response  to  the  LFT  law,  the  position  of  Director,  Live  Fire  Testing,  was  established 
under  the  Office  of  the  Deputy  Director  of  Defense  Research  and  Engineering  (Test  and 
Evaluation)  [DDDR&E(T&E)]  and  was  given  the  responsibility  for  implementing  the  law. 
The  Live  Fire  Test  &  Evaluation  (LFT&E)  Guidelines  were  written  in  1988  by  the  Test  and 
Evaluation  Committee,  and  the  LFT&E  Planning  Guide  was  written  in  1989  by  the  Director. 
LFT,  to  assist  the  system  program  managers  in  the  preparation  of  test  plans  that  satisfy  the 
law.  In  spite  of  these  efforts,  the  historical  reluctance  to  conduct  realistic  vulnerability  tests 
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on  aircraft  systems  in  development  has  persisted.  Furthermore,  ambiguities  in  the  wording 
of  the  law,  and  in  the  published  guidance  for  satisfying  the  law,  have  resulted  in  different 
interpretations  of  the  law’s  requirements  regarding  what  must  be  tested  and  what  weapons 
must  be  used.  Considerable  controversy  between  the  LFT  Office  and  the  Services  has 
developed  as  a  consequence  of  these  different  interpretations  of  the  law’s  requirements. 

In  an  attempt  to  resolve  the  controversy  regarding  the  law’s  requirements,  and  to  obtain 
an  independent  opinion  regarding  the  total  vulnerability  assessment  process  and  the  law’s, 
contribution  to  that  process,  the  DDDR&E(T&E),  now  the  Director,  Test  and  Evaluation, 
requested  that  the  National  Research  Council  (NRC)  conduct  a  study  of  aircraft  vulnerability 
assessment.  The  NRC  appointed  the  Committee  on  Weapons  Effects  on  Airborne  Systems 
to  conduct  this  study.  This  report  is  the  result  of  the  committee’s  deliberations. 

The  committee  was  asked  to  review  the  two  methodologies  used  to  assess  the  vulnerability 
of  airborne  systems  (i.e.,  analysis/modeling  and  live  fire  testing)  and  to  identify  and  evaluate 
the  costs  and  effectiveness  of  these  methodologies.  Recommendations  regarding  the  most 
appropriate  methodology  for  each  application  were  requested.  In  particular,  the  current 
direction  of  the  congressionally  mandated  LFT&E  program  was  to  be  reviewed  and 
recommendations  for  change  were  requested,  if  appropriate. 

The  tasks  assigned  to  the  committee  were  carried  out  by  examining  the  two  aircraft 
vulnerability  assessment  methodologies  and  the  LFT  law  from  the  general  point  of  view  that 
the  purpose  of  any  assessment  and  of  the  law  is  to  obtain  information  on  the  vulnerability 
of  aircraft.  This  information  can  be  described  in  terms  of  the  information  attributes  of  (1) 
types,  amounts,  and  applications;  (2)  the  accuracy  of,  or  level  of  confidence  in,  the  information; 
and  (3)  the  cost  required  to  obtain  the  information.  In  addition,  the  LFT  law  was  considered 
to  be  a  mandated  application  of  the  live  fire  test  methodology.  Accordingly,  Chapter  1 
presents  a  review  of  the  two  methodologies  and  identifies  the  applications  of  the  results 
from  these  methodologies.  Chapter  2  presents  an  evaluation  of  the  cost,  effectiveness,  and 
deficiencies  of  the  two  methodologies  based  upon  the  three  information  attributes  described 
above.  Chapter  3  presents  a  review  of  the  LFT&E  program  directed  by  the  LFT  Office. 
Chapter  4  contains  the  LFT  programs  of  the  three  Services  and  the  views  of  the  Services’ 
LFT  test  community  and  of  industry.  Chapter  5  presents  the  committee’s  view  of  the  future 
of  vulnerability  assessment,  and  Chapter  6  gives  the  committee’s  conclusions  and 
recommendations.  The  LFT  law  and  the  major  issues  and  conclusions  of  two  earlier  studies 
of  vulnerability  assessment  and  live  fire  testing  of  military  vehicles  by  the  U.S.  General 
Accounting  Office  and  the  Board  on  Army  Science  and  Technology  are  presented  in 
Appendixes  A,  B,  and  C,  respectively.  The  committee  believes  that  its  recommendations 
merit  serious  consideration  and  that  the  results  of  this  study  should  eliminate,  or  at  least 
significantly  reduce,  the  controversy  that  has  revolved  around  the  LFT  law. 

The  committee,  in  its  deliberations,  discovered  that  the  controversy  over  the  LFT  law 
was  aggravated  by  several  inconsistent  definitions  of  important  words  and  terms  used  in 
vulnerability  assessment  Consequently,  the  committee  has  included  the  various  published 
definitions  as  well  as  those  selected  for  use  in  this  report  in  the  following  chapter  entitled 
“Definitions.**  The  key  definitions  used  in  the  report  are  live  fire  test  (any  test  that  involves 
the  firing  of  actual  munitions  at  a  target);  Live  Fire  Test  (a  live  fire  test  that  is  part  of 
the  congressionally  mandated  LFT&E  program);  full-scale  or  complete  system  test  (a  test 
conducted  on  the  complete  or  total  system,  with  or  without  the  full  complement  of  fuel, 
ammunition,  and  hydraulic  fluid  carried  into  combat);  sub-scale  or  partial  system  test  (a  test 
conducted  on  a  part  of  the  system,  such  as  a  component,  a  subsystem,  or  a  subassembly, 
with  or  without  the  full  complement  of  fuel,  ammunition,  and  hydraulic  fluid  carried  into 
combat);  and  full-up  test  (a  test  conducted  on  a  complete  or  a  partial  system  with  the  full 
complement  of  fuel,  ammunition,  and  hydraulic  fluid  carried  by  the  system  into  combat). 

The  committee  held  four  meetings.  The  first  meeting  was  in  July  1991  in  Washington, 
D.C.  The  committee  received  briefings  from  the  study  sponsor,  the  Live  Fire  Test  Office, 
the  Tactical  Warfare  Programs  Office,  the  Joint  Live  Fire  Test  Program  Office,  and  the 


aircraft  vulnerability  analysis/modeling  community.  This  was  followed  by  a  meeting  in 
September  1991  in  Washington,  D.C.,  when  the  committee  heard  briefings  from  the  .Army, 
Navy,  and  Air  Force  Live  Fire  Test  officials,  the  Ballistic  Research  Laboratory,  the  Institute 
for  Defense  Analyses,  and  the  program  offices  of  several  major  systems.  In  January  1992, 
the  committee  met  at  the  Arnold  and  Mabel  Beckman  Center  in  Irvine,  California,  and  heard 
briefings  from  several  industry  representatives  involved  with  Live  Fire  Testing  and  from  the 
Director,  Live  Fire  Testing.  During  this  month,  several  of  the  committee  members  also  met 
with  the  congressional  staffer  who  drafted  the  LFT  law.  The  committee  conducted  a  report 
writing  session  in  April  1992. 

The  individuals  who  contributed  to  the  work  of  the  committee  during  the  course  of  this 
study  are  too  numerous  to  be  listed  separately  here.  However,  in  addition  to  the  initial 
guidance  provided  by  Mr.  Charles  Adolph,  the  study  sponsor,  the  committee  would  like  to 
acknowledge  with  gratitude  the  assistance  received  from  personnel  from  the  Live  Fire  Test 
Office,  the  Joint  Live  Fire  Test  Program  Office,  the  Institute  for  Defense  Analyses,  the 
Ballistic  Research  Laboratory,  the  Army/Navy/Air  Force  Live  Fire  Test  Offices  and  major 
Program  Offices,  and  representatives  from  Boeing,  General  Dynamics,  McDonnell-Douglas, 
and  Northrop.  The  committee  also  wishes  to  thank  the  staff  of  the  Air  Force  Studies  Board 
and  the  sponsor  liaison,  COL  Bernard  (Chip)  Ferguson,  for  providing  excellent  support 
throughout  this  study. 

Robert  E.  Ball 

Chairman 
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Definitions 


Many  of  the  important  words  and  terms  used  in  the  survivability,  lethality,  and  live  fire 
test  disciplines  have  been  defined  in  several  public  documents.  The  published  definitions  of 
the  words  and  terms  of  particular  interest  to  this  study  are  given  in  the  Table  below. 


Previously  Published  Definitions 

configured  for  combat 

The  term  “configured  for  combat,”  with  respect  to  a 
weapon  system,  platform,  or  vehicle,  means  loaded  or 
equipped  with  all  dangerous  materials  (including  all 
flammables  and  explosives)  that  would  normally  be 
on  board  in  combat  (U.S.  Congress,  1986-1989). 

covered  product  improvement  program 

A  program  under  which  a  modification  will  be  made 
to  a  covered  system  that  (as  determined  by  the  Secre¬ 
tary  of  Defense)  is  likely  to  affect  significantly  the 
survivability  of  such  system  (U.S.  Congress,  1986- 
1989). 

covered  system 

A  vehicle,  weapon  platform,  or  conventional  weapon 
system  that  includes  features  designed  to  provide  some 
degree  of  protection  to  users  in  combat  and  is  a  major 
system  (see  section  2303(5)  of  title  10  U.S.  Code  for 
the  definition  of  a  major  system)  (U.S.  Congress,  1986- 
1989). 

damage  mechanism 

The  output  of  a  warhead  that  causes  damage  to  the 
aircraft  (Balt  1985). 

damage  process 

The  interaction  between  the  damage  mechanism  and 
the  aircraft  components  (Balt  1985). 
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full-scale  tests 

Those  conducted  on  complete  weapons  systems  rather 
than  components  or  mock-ups  (GAO.  1987). 

full-up  testing 

Firings  against  a  full-scale  target  containing  all  the 
dangerous  materials  (e.g.,  ammunition,  fuel,  hydraulic 
fluids),  system  parts  (e.g.,  electrical  lines  with  operat¬ 
ing  voltages  and  currents  applied,  hydraulic  lines  con¬ 
taining  appropriate  fluids  at  operating  pressures),  and 
stowage  items  normally  found  on  that  target  when 
operating  in  combat.  Full-up  testing  includes  firings 
against  full-up  components,  full-up  subsystems,  full- 
up  subassemblies,  or  full-up  systems.  The  term  “full- 
up  testing”  is  synonymous  with  “realistic  survivability 
testing”  or  “realistic  lethality  testing”  as  defined  in 
the  legislation  covering  Live  Fire  Testing  (OSD.  1988; 
DDDR&E;  1989). 

full-up  tests 

Those  conducted  with  the  full  complement  of  fuel, 
ammunition,  and  hydraulic  fluid  carried  by  the  system 
into  combat  (GAO,  1987). 

Live  Fire  Test 

A  test  event  within  an  overall  Live  Fire  Test  &  Evalua¬ 
tion  program  that  involves  the  firing  of  actual  munitions 
at  target  components,  target  subsystems,  target  subas¬ 
semblies,  and/or  full-scale  targets  to  examine  personnel 
casualty,  vulnerability,  and/or  lethality  issues  (OSD, 
1988:  DDDR&E,  1989). 

Live  Fire  Test  and  Evaluation 
(LFT&E)  Program 

The  program  conducted  by  the  Director,  Live  Fire 
Testing,  Office  of  the  Director  of  Defense  Research 
and  Engineering  and  described  in  (DDDR&E,  1989). 

Live  Fire  Test  law 

Legislation  in  the  fiscal  year  1987  Department  of  De¬ 
fense  Authorization  Act  that  requires  realistic  surviv¬ 
ability  testing  before  a  covered  system  can  proceed 
beyond  low-rate  initial  production  (U.S.  Congress, 
1986-1989). 

realistic  survivability  testing 

The  term  “realistic  survivability  testing”  means  test¬ 
ing  for  vulnerability  of  the  system  in  combat  by  firing 
munitions  likely  to  be  encountered  in  combat  (or  muni¬ 
tions  with  a  capability  similar  to  such  munitions)  at 
the  system  configured  for  combat,  with  primary  empha¬ 
sis  on  testing  vulnerability  with  respect  to  potential 
user  casualties  and  by  taking  into  equal  consideration 
the  susceptibility  to  attack  and  combat  performance  of 
the  system  (U.S.  Congress.  1986-1989). 

simulants 

Fabricated  substitutes  for  unavailable  threat  weapons 
or  targets  (GAO,  1987). 

sub-scale  tests 

Any  tests  conducted  on  less-than-fuil-scale  target 
weapon  systems,  such  as  component  vulnerability  tests 
or  be  hind-armor-debris  studies  (GAO,  1987). 

surrogate 

Any  existing  munition  or  target  substituted  for  one  that 
is  unavailable  for  testing  on  the  basis  of  similarity 
(GAO.  1987). 

survivability 

The  capability  of  an  aircraft  to  avoid  and/or  withstand 
a  man-made  hostile  environment  (Ball.  1985). 
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The  ability  of  a  weapon  system  to  avoid  being  killed 
in  battle,  including  its  vulnerability  if  hit.  but  also 
taking  other  factors  such  as  maneuverability  and  the 
ability  to  avoid  detection  into  account  (GAO.  1987). 


susceptibility 

The  inability  of  an  aircraft  to  avoid  the  radars,  guns, 
ballistic  projectiles,  guided  missiles,  exploding  war¬ 
heads.  and  other  elements  that  make  up  the  hostile 
environment.  It  can  be  measured  in  a  general  sense  by 

PM,  the  probability  the  aircraft  is  hit  by  one  or  more 
damage -causing  mechanisms,  such  as  a  bullet,  frag¬ 
ment.  or  blast  wave  (Ball,  1985). 

Comprises  all  the  capabilities  and  characteristics  of  a 
target  and  threat  that  influence  or  determine  the  proba¬ 
bility  that  the  target  is  hit.  including  the  threat  capability 
to  detect  lock  on.  track,  and  fire,  and  the  target  capabil¬ 
ity  to  evade  the  threat  (GAO,  1987). 

vulnerability 

The  inability  of  an  aircraft  to  withstand  the  damage 
caused  by  the  hostile  environment  Vulnerability  can 
be  measured  by  P**,  the  conditional  probability  that 
the  aircraft  is  killed  given  a  hit  by  a  damage  mecha¬ 
nism.  such  as  a  bullet  or  fragment  or  by  P^o,  die 
conditional  probability  that  the  aircraft  is  killed  by  a 
warhead  detonation  (Ball,  1985). 

The  inability  of  a  weapon  system  to  withstand  damage 
from  a  specific  attack,  given  that  it  has  been  hit  (GAO. 
1987). 

The  committee  notes  that  the  definitions  for  several  of  the  words  and  terms  given 
above  either  are  not  consistent  or  do  not  conform  with  conventional  usage.  This  inconsistency 
and  lack  of  conformity  has  created  considerable  confusion.  The  committee  has  reviewed 
these  definitions  and  has  selected  the  definitions  given  below  as  the  ones  to  use  throughout 
this  report. 


Definitions  Used  in  This  Report 

live  fire  test 

Any  test  that  involves  the  firing  of  actual  munitions 
at  a  target. 

Live  Fire  Test 

A  live  fire  test  that  is  a  pan  of  the 
congressionally  mandated  LFT&E  program. 

full-scale  or  complete  system  test 

A  test  conducted  on  the  complete  or  total  system,  with 
or  without  the  full  complement  of  fuel,  ammunition, 
and  hydraulic  fluid  carried  into  combat 

sub-scale  or  partial  system  test 

A  test  conducted  on  a  pan  of  the  system,  such  as  a 
component,  a  subsystem,  or  a  subassembly,  with  or 
without  the  full  complement  of  fuel,  ammunition,  and 
hydraulic  fluid  carried  into  combat 

full-up  test 

A  test  conducted  on  a  complete  system  or  a  partial 
system,  with  the  full  complement  of  fuel,  ammunition, 
and  hydraulic  fluid  carried  by  the  system  into  combat 

DEFINITIONS 


xvu 


men.  test 


A  test  conducted  on  a  complete  system  or  a  partial 
system,  without  the  full  complement  of  fuel,  ammuni¬ 
tion,  and  hydraulic  fluid  carried  by  the  system  into 
combaL  A  semi-inert  test  is  one  in  which  some  of  the 
combustibles  are  on-board. 


Note  the  important  distinction  between  a  live  fire  test  and  a  Live  Fire  Test.  Also  note 
that  a  “complete  system  test"  is  equivalent  to  a  “full-scale  test/’  and  a  “partial  system 
test”  is  equivalent  to  a  “sub-scale  test.”  Furthermore,  the  term  “full-up  test”  does  not 
imply  that  a  complete  system  is  tested. 
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Statement  of  Task 


There  is  concern  by  the  Director,  Test  and  Evaluation,  for  evaluating  airborne  systems  level 
vulnerability  assessment  methodologies.  Several  methodologies  such  as  computer  modeling, 
engineering  analysis,  and  live-fire  testing  are  available  to  the  Services. 

The  committee  will: 

1.  review  current  methodologies  used  by  the  Army,  Navy,  and  Air  Force  to  determine 
the  vulnerability  of  airborne  systems  to  enemy  conventional  weapons  and  identify  the  applica¬ 
tions  of  the  results; 

2.  evaluate  the  costs  and  the  effectiveness  of  these  methodologies  and  identify  deficiencies; 

3.  recommend  the  most  appropriate  methodologies  for  the  applications,  weighing  the 
confidence  in  the  results  versus  costs; 

4.  in  particular,  review  the  current  direction  on  live  fire  testing  and  evaluation,  compare 
this  direction  to  the  committee’s  recommended  methodologies,  and  recommend  changes,  if 
appropriate. 
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EXECUTIVE  SUMMARY 


Background 

Aircraft  vulnerability  refers  to  the  inability  of  the  aircraft  to  withstand  the  damage  caused 
by  one  or  more  hits  by  the  damage  mechanisms  associated  with  air  defense  weapons,  such 
as  warhead  fragments  and  blast.  Vulnerability  assessment  or  evaluation  is  a  part  of  every 
U.S.  military  aircraft  acquisition  program.  The  assessment  can  be  accomplished  by  using 
two  methodologies:  (1)  analyses  or  computer  models  that  simulate  the  various  reactions  of 
the  aircraft  and  its  components  to  the  hits,  and  (2)  live  fire  testing. 


The  Live  Fire  Test  Law.  As  a  result  of  the  controversy  over  the  vulnerability  testing  of 
the  U.S.  Army’s  Bradley  Fighting  Vehicle,  Congress  passed  a  law  in  fiscal  year  (FY)  1987, 
known  as  the  Live  Fire  Test  (LFT)  law,  which  mandates  realistic  survivability  and  lethality 
testing  of  covered  systems  or  programs.  The  law  was  modified  in  1988.  In  the  current  version 
of  the  law,  realistic  survivability  testing  is  defined  as  “testing  for  vulnerability  of  the  system 
in  combat  by  firing  munitions  likely  to  be  encountered  in  combat  (or  munitions  with  a 
capability  similar  to  such  munitions)  at  the  system,  configured  for  combat,  with  the  primary 
emphasis  on  testing  vulnerability  with  respect  to  potential  user  casualties  and  taking  into 
equal  consideration  the  susceptibility  to  attack  and  combat  performance  of  the  system.” 
(Note  that  survivability  is  used  when  vulnerability  is  intended) 

According  to  the  LFT  Guidelines  in  the  law,  the  live  fire  tests  are  to  be  carried  out 
sufficiendy  early  in  the  development  phase  of  the  system  to  allow  any  design  deficiency 
demonstrated  by  the  testing  to  be  corrected  in  the  design  of  the  system  before  proceeding 
beyond  low-rate  initial  production  (LRIP).  The  system  acquisition  program  cannot  proceed 
beyond  LRIP  until  the  testing  is  completed  The  FY1988-1989  Department  of  Defense 
Authorization  Act  Conference  Report  states  that  Congress  intended  that  the  Secretary  of 
Defense  implement  the  LFT  law  “in  a  manner  which  encourages  the  conduct  of  full-up 
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vulnerability  and  lethality  tests  under  realistic  combat  conditions,  first  at  the  sub-scale  level 
as  they  are  developed,  and  later  at  the  full-scale  level  mandated  in  the  legislation." 

A  waiver  from  the  requirement  for  realistic  survivability  testing  is  allowed  if  the  Secretary 
of  Defense  notifies  Congress,  prior  to  entry  into  full-scale  engineering  development,  that  live 
fire  testing  of  the  system  would  be  unreasonably  expensive  and  impractical.  The  notification  of 
the  waiver  must  be  accompanied  by  a  report  describing  an  alternate  program  for  evaluating 
the  survivability  of  the  system  and  assessing  possible  alternatives  to  realistic  survivability 
testing  of  the  system. 

When  the  Live  Fire  Test  law  was  passed,  the  position  of  Director,  Live  Fire  Testing,  was 
established  under  the  Office  of  the  Deputy  Director,  Defense  Research  and  Engineering  (Test 
and  Evaluation)  [DDDR&E(T&E)]  and  was  given  the  responsibility  for  implementing  the 
LFT  legislation.  The  tests  specifically  associated  with  the  congressionally  mandated  Live 
Fire  Test  and  Evaluation  (LFT&E)  program  are  referred  to  herein  as  Live  Fire  Tests,  whereas 
other  tests  using  live  ammunition  that  are  not  specifically  part  of  the  LFT&E  program  are 
referred  to  as  live  fire  tests. 


Study  Tasks .  The  tasks  assigned  to  this  committee  were  (1)  to  review  and  evaluate  the 
current  vulnerability  assessment  methodologies  for  aircraft,  including  both  analysis/modeling 
and  live  fire  testing;  (2)  to  review  and  evaluate  the  current  direction  of  the  congressionally 
mandated  Live  Fire  Test  programs  within  OSD  and  the  Services;  and  (3)  to  recommend 
changes  to  these  methodologies  and  programs,  if  appropriate.  The  committee  was  instructed 
to  consider  all  aspects  of  the  vulnerability  assessment  methodologies,  programs,  and  the  Live 
Fire  Test  legislation. 

The  committee  met  four  times  between  July  1991  and  April  1992  and  received  presentations 
from  personnel  from  the  Office  of  the  Secretary  of  Defense  (OSD)  LFT&E  Office,  the 
DDDR&E(T&E)/Tactical  Weapons  Programs  Office,  the  U.S.  Army’s  Test  and  Evaluation 
Management  Agency  and  Comanche  Program  Office,  the  U.S.  Navy’s  Survivability  Branch, 
Naval  Air  Systems  Command,  the  U.S.  Air  Force’s  Test  and  Evaluation  Office  and  F-22 
and  C-17  System  Program  Offices,  and  the  live  fire  test  organizations  of  the  three  Services, 
as  well  as  vulnerability  assessment  experts  from  four  U.S.  aircraft  companies  and  the  Institute 
for  Defense  Analyses.  Several  committee  members  also  interviewed  Mr.  Joseph  Cirincione, 
the  congressional  staff  member  who  drafted  the  LFT  legislation  in  1987. 

The  study  was  divided  into  six  parts:  (1)  review  of  the  analysis/modeling  and  live  fire 
testing  assessment  methodologies  and  identification  of  the  applications  of  the  results  of  the 
assessments;  (2)  evaluation  of  the  cost,  effectiveness,  and  deficiencies  of  the  two  methodolo¬ 
gies:  (3)  review  and  evaluation  of  the  OSD  LFT&E  program;  (4)  review  and  evaluation  of 
the  LFT&E  programs  of  the  three  Services;  (5)  examination  of  the  future  of  vulnerability 
assessment  of  aircraft;  and  (6)  conclusions  and  recommendations. 


Review  and  Evaluation 

Methodology  Review  and  Applications  of  the  Results .  The  study  reviewed  both  of  the 
methodologies  used  by  the  Army,  Navy,  and  Air  Force  to  determine  the  vulnerability  of 
airborne  systems  to  guns  and  guided  missiles,  and  identified  the  applications  of  the  results.  The 
weapons  considered  include  the  nonexpiosive  armor-piercing  ( AP)  penetrator  or  fragment,  the 
contact-fuzed  high-explosive  (HE)  warhead,  and  the  proximity-fuzed  externally  detonating 
HE  warhead.  The  target  aircraft  is  either  full-scale  (the  complete  system)  or  sub-scale  (a 
partial  system  consisting  of  one  or  more  components  and/or  subsystems)  and  inert  (no 
combustibles)  or  full-up  (with  combustibles). 

The  committee  identified  six  applications  for  the  results  from  analysis/modeling  and  live 
fire  testing.  They  are  (1)  to  aid  in  the  design  and  design  validation,  (2)  to  satisfy  the 
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vulnerability  assessment  requirements  contained  in  DoD  MIL-STD  2069.'  (3)  to  develop 
data  bases  in  support  of  subsequent  analytical  assessments,  (4)  to  predict  test  outcomes,  (5) 
to  satisfy  the  requirements  of  the  Live  Fire  Test  law,  and  (6)  to  support  acquisition  decisions. 


Cost ,  Effectiveness ,  and  Deficiencies  of  the  Methodologies .  The  committee  notes  that  the 
primary  objectives  and  applications  of  the  two  methodologies  are,  in  general,  different. 
Analysis  and  modeling  are  primarily  used  to  aid  in  design  and  to  quantify  vulnerability, 
whereas  live  fire  testing  is  conducted  to  gain  insight  into  the  major  types  of  physical  damage, 
including  cascading  and  synergistic  effects;  to  develop  component  vulnerability  data  bases; 
to  aid  in  design;  and  to  validate  the  design  and  the  model,  when  appropriate. 

The  committee  selected  the  information  or  results  provided  by  the  methodologies  as  the 
basis  for  comparison  with  respect  to  cost,  effectiveness,  and  deficiencies.  In  particular,  the 
information  attributes  of  (1)  type,  amount,  and  applications;  (2)  accuracy,  or  level  of  confi¬ 
dence;  and  (3)  cost  were  evaluated  for  both  analysis/modeling  and  testing.  The  testing 
methodology  was  divided  into  two  categories,  tests  on  sub-scale  targets  and  tests  on  full- 
scale  targets. 

In  general,  the  results  from  analyses  and  models  consist  of  numerical  values  of  vulnerability 
for  the  components  and  the  aircraft  for  all  weapons  and  ail  threat  directions.  These  results 
can  be  used  in  all  six  applications  listed  above.  The  overall  level  of  confidence  in  the 
analytical  results  is  relatively  low  because  of  the  inadequate  modeling  and  supporting  data 
base  of  some  damage  processes  and  vulnerabilities,  and  the  omission  of  others.  The  cost  of 
analysis/modeling  is  also  relatively  low. 

The  live  fire  tests  on  both  sub-scale  and  full-scale  targets  produce  information  on  what 
actually  happened  for  a  particular  set  of  test  conditions  but  only  for  a  relatively  small  number 
of  shots  under  these  conditions.  The  applications  of  the  test  results  are  to  aid  in  design  and 
design  validation,  to  develop  vulnerability  data  bases,  to  satisfy  the  Live  Fire  Test  law,  and 
to  support  acquisition  decisions.  The  level  of  confidence  in  the  results  is  relatively  high,  and 
so  is  the  cost,  particularly  for  the  full-scale  aircraft  tests.  However,  using  other  full-scale 
test  articles,  such  as  prototypes,  for  the  full-scale  vulnerability  testing  can  significantly  reduce 
the  cost.  Any  Live  Fire  Tests  on  prototypes  must  be  carefully  audited  for  their  applicability 
to  production  articles  since  the  differences  between  the  two  may  be  large. 

One  of  the  most  important  findings  of  this  study  is  that  on-board  ordnance  has  been 
neglected  in  both  analyses  and  live  fire  testing  as  a  contributor  to  vulnerability.  One  of  the 
basic  requirements  of  the  Live  Fire  Test  program  is  to  test  full-scale  vehicles  with  the  full 
load  of  on-board  ordnance.  External  ordnance  may  shield  components  from  projectiles  and 
fragments,  or  it  may  react  violently  to  a  ballistic  impact,  possibly  destroying  the  aircraft. 
Adverse  reactions  of  any  internally  carried  ordnance  have  an  even  greater  probability  of 
destroying  the  aircraft. 

In  general,  the  committee  believes  that  the  combination  of  analytical  models,  supported 
by  live  fire  tests  on  components  and  subsystems,  and  the  full-scale  Live  Fire  Tests  are 
mutually  compatible  in  the  vulnerability  assessment  and  design  of  aircraft.  They  complement 
each  other,  and  the  whole  is  superior  to  the  sum  of  the  parts.  However,  more  work  is  needed 
to  unify  these  approaches  in  order  to  obtain  the  maximum  benefit 

Review  of  the  Requirements  of  the  Live  Fire  Test  Law .  The  committee  notes  that  there 
is  a  major  controversy  among  the  various  participants  in  the  DoD  Live  Fire  Test  program 
regarding  the  law’s  requirements.  One  point  of  contention  is  the  requirement  for  testing  the 
system.  One  interpretation  is  that  the  law  does  not  explicitly  state  that  a  complete  system 
must  be  tested;  hence  the  law  is  satisfied  by  an  LFT  program  only  on  components  and 


1  DoD  MIL-STD  2069.  “Requirements  for  Aircraft  Nonnuclear  Survivability  Program,"  contains  the  requirements 
and  guidelines  for  establishing  and  conducting  aircraft  survivability  programs. 
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subsystems.  Another  interpretation  is  that  the  word  “system”  refers  to  the  complete  or  full- 
scale  system.  Based  upon  the  evidence  gathered  by  the  committee  and  its  study  of  the  law, 
the  committee  is  unanimous  in  the  opinion  that  the  LFT  law  requires  a  full-scale,  full-up 
aircraft  to  be  tested,  unless  a  waiver  is  granted.  The  committee  bases  its  opinion  upon  the 
events  that  led  to  the  law,  the  wording  in  the  law,  the  accompanying  discussion  of  the 
law  in  the  FY 1 988- 1 989  DoD  Authorization  Act  Conference  Report,  the  opinion  of  the 
congressional  staff  member  who  drafted  the  law,  and  the  fact  that  a  waiver  is  allowed.  If 
full-scale,  full-up  tests  were  not  required,  no  waiver  would  be  necessary,  and  any  live  fire 
tests  would  suffice,  provided  they  were  realistic. 


Review  of  the  OSD  Live  Fire  Test  and  Evaluation  Program .  Two  OSD  documents  have 
been  provided  to  the  Services  and  their  Program  Managers  to  assist  them  in  the  planning 
and  conduct  of  a  Live  Fire  Test  program:  (1)  the  1988  Live  Fire  Test  and  Evaluation 
Guidelines  issued  by  the  Test  and  Evaluation  Committee,  Office  of  the  Secretary  of  Defense, 
and  (2)  the  1989  Live  Fire  Test  and  Evaluation  Planning  Guide  issued  by  the  Live  Fire  Test 
Office.  The  committee  is  concerned  that  the  written  guidance  provided  by  the  LFT&E  Office 
does  not  provide  sufficient  detail,  particularly  with  respect  to  the  full-scale  tests,  to  ensure 
that  the  Program  Manager  can  satisfy  the  requirements  of  the  OSD  policy  and  the  LFT  law, 
and  also  design  a  cost-effective  test  plan  that  will  ensure  the  system  requirements  are  satisfied. 
The  various  definitions  given  in  the  1988  LFT&E  Guidelines  have  been  interpreted  by  some 
to  imply  that  full-scale,  fuil-up  Live  Fire  Tests  do  not  have  to  be  conducted,  that  is,  the  OSD 
LFT  policy  is  satisfied  by  Live  Fire  Tests  only  on  sub-scale  targets,  such  as  major  portions 
or  subassemblies  of  an  aircraft.  Furthermore,  the  committee  is  concerned  about  the  official 
status,  or  lack  thereof,  of  the  LFT&E  Guidelines  and  the  Planning  Guide. 


Review  of  the  Service  Live  Fire  Test  and  Evaluation  Programs .  The  committee  reviewed 
the  LFT&E  policies  and  programs  of  the  three  Services,  including  the  Army’s  RAH-66,  the 
Navy’s  V-22  and  A-12,  and  the  Air  Force’s  F-22  and  C-17.  The  committee  believes  it  is 
important  to  point  out  the  facts  that,  except  for  the  RAH-66,  all  of  these  programs  were  well 
under  way  when  the  law  was  passed  and  the  C-17  development  had  proceeded  beyond  the 
deadline  for  the  application  of  a  waiver.  Furthermore,  Congress  did  not  provide  transitional 
guidance  for  these  programs  or  additional  money  to  fund  the  tests. 

The  Navy  and  the  Air  Force  have  interpreted  the  1988  LFT&E  Guidelines  to  imply  that 
full-scale,  full-up  tests  are  not  required.  Furthermore,  the  LFT&E  policies  presented  to  the 
committee  by  the  Services  do  not  consider  such  tests  to  be  cost-effective,  particularly  if  on¬ 
board  ordnance  is  included.  Consequently,  they  have  not  developed  LFT&E  programs  that 
contain  full-scale,  fuil-up  Live  Fire  Tests.  However,  both  Services  strongly  support  the 
conduct  of  sub-scale  inert  and  full-up  tests  throughout  the  development  process.  The  Army 
policy  on  LFT&E  supports  a  “building-block”  approach  consisting  of  component  testing 
through  full-scale,  full-up  system  testing  that  satisfies  the  Live  Fire  Test  law.  The  emphasis 
of  its  LFT&E  program  is  on  sub-scale  testing,  with  limited  full-scale,  full-up  testing  to 
confirm  the  results  obtained  from  sub-scale  testing.  However,  the  Army  LFT&E  program 
for  the  RAH-66  did  not  contain  firm  plans  for  testing  a  full-scale,  full-up  helicopter.  The 
full-scale  testing  was  going  to  be  conducted  only  if  the  sub-scale  test  results  showed  it  to 
be  necessary. 

In  addition  to  the  controversy  regarding  the  requirement  for  full-scale  testing,  there  is  a 
controversy  regarding  the  munitions  to  be  used  in  the  Live  Fire  Tests.  The  specific  munitions 
to  be  used  for  a  particular  aircraft  are  selected  by  the  aircraft  Program  Office  as  part  of  its 
LFT&E  program  plan.  Typically,  the  threats  selected  by  the  Program  Office  for  Live  Fire 
Testing  are  the  threats  the  aircraft  was  designed  to  withstand,  such  as  a  single  hit  by  an  AP 
projectile  or  small-caliber  HE  round.  The  assumption  is  made  that  the  more  lethal  overmatch¬ 
ing  threats,  such  as  the  larger  gun  projectiles  and  missiles,  will  be  avoided  and  hence  should 
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not  be  a  pan  of  the  LFT&E  program.  The  Live  Fire  Test  Office  has  interpreted  the  phrase 
“munitions  likely  to  be  encountered  in  combat”  to  mean  that  those  munitions  the  aircraft 
may  encounter,  including  the  latest  directed  energy  weapons,  should  be  included  in  the 
LFT&E  program,  regardless  of  the  design  threat  for  the  aircraft. 

The  representatives  from  the  live  fire  test  organizations  of  the  three  Services  and  the  • 
vulnerability  experts  from  industry  appear  to  be  in  general  agreement  about  the  efficacy  of 
the  Live  Fire  Test  law  as  it  applies  to  aircraft.  They  do  not  consider  it  necessary  to  conduct 
full-scale,  full-up  tests  in  order  to  determine  most  of  the  design  vulnerabilities.  However, 
they  do  consider  it  essential  to  conduct  many  sub-scale  live  fire  tests  on  components  and 
sub-systems,  both  inert  and  full-up,  in  the  development  cycle  of  an  aircraft.  They  believe 
the  full-scale,  full-up  tests  may  be  conducted  too  late  in  the  development  cycles  to  be  of 
much  value  to  the  designer,  and  that  the  amount  of  information  obtained  from  the  tests  is 
limited.  The  representatives  do  recognize  the  possibility  of  an  unanticipated  reaction,  cascad¬ 
ing  damage,  or  synergism  occurring  in  the  full-scale  aircraft.  However,  they  believe  that 
nearly  all  of  the  kill  modes  of  an  aircraft  are  known  and  can  be  anticipated. 

The  committee  notes  that  not  everyone  who  has  observed  live  fire  tests  on  sub-scale  and 
full-scale  test  articles  shares  the  views  held  by  these  testers  and  vulnerability  experts.  They 
believe  that  there  have  been  unanticipated  results  from  these  tests.  Furthermore,  even  when 
the  response  is  as  expected,  the  difference  in  the  expected  magnitude  of  the  response  and 
the  observed  magnitude  often  is  too  large  to  be  acceptable.  The  testers  have  also  overlooked 
the  fact  that  the  information  from  the  full-scale  tests  is  a  valuable  input  to  the  acquisition 
decision  makers  at  milestone  reviews. 

Perceptions  of  the  LFT  Law .  The  committee  is  aware  of  the  strong  differences  of  opinion 
held  by  various  individuals  and  organizations  concerning  the  efficacy  of  the  Live  Fire  Test 
law  and  of  the  level  of  mutual  distrust  that  has  evolved  as  a  result  of  these  opinions.  This 
distrust  between  the  various  participants  of  each  other’s  motives  and  actions  is  probably 
responsible  for  the  ever-increasing  tensions  within  the  current  Live  Fire  Test  program.  In 
the  committee’s  opinion,  the  attitudes  of  the  major  participants  concerning  the  Live  Fire 
Test  law  and  its  place  in  the  acquisition  process  may  be  those  described  below. 

In  the  committee’s  opinion,  the  Program  Manager  (PM)  may  consider  the  full-scale,  full- 
up  testing  mandated  by  the  LFT  law  to  be  an  unquantifiable,  but  potentially  catastrophic, 
risk  to  his  program.  LFT  has  no  quantitative  contractual  specifications  or  acceptance  criteria 
at  program  initiation.  No  quantitative  criteria  for  acceptable  or  unacceptable  damage  are 
included  in  the  requirements  process,  milestone  commitments,  or  contractor  performance 
documents.  LFT  of  the  full-scale  aircraft  occurs  late  in  the  development  phase  of  the  program, 
and  there  may  be  neither  adequate  time  nor  money  to  conduct  the  tests  or  to  make  any 
changes  required  as  a  result  of  the  tests.  Further,  the  PM  may  believe  that  neither  the  definition 
of  the  tests,  nor  the  conduct  of  the  tests,  nor  the  interpretation  of  the  test  results  is  totally 
under  his  control.  The  perceived  jeopardy  to  his  program  created  by  LFT  is  exacerbated  by 
the  severe  requirement  to  fit  the  program  into  a  somewhat  inflexible  overall  resource  schedule; 
both  in  time  and  in  dollars.  In  summary,  LFT  represents  a  considerable  source  of  problems 
to  the  PM,  in  the  form  of  an  uncontrollable,  potentially  catastrophic  uncertainty,  as  he 
attempts  to  successfully  complete  the  development  of  a  system,  and  should  be  avoided  if  at 
all  possible. 

In  the  committee’s  opinion,  the  Services  may  believe  that,  as  system  developers  and  users, 
they  know  what  is  needed  in  the  equipment  they  will  take  into  the  field,  and  that  they,  the 
Services,  are  directly  responsible  for  the  fate  of  the  military  personnel  who  use  this  equipment 
in  combat  They  are  very  apprehensive  about  any  outside  organization  that  can  dilute  their 
ability  to  define  the  necessary  equipment  testing  and  the  procedures  required  to  accomplish 
this  testing.  They  appear  to  further  believe  that  the  Live  Fire  Test  law  gives  to  others  not 
directly  responsible  for  the  delivered  product  inordinate  control  without  any  accompanying 
responsibility  for  the  quality  of  the  product  or  its  cost 
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In  the  committee’s  opinion,  the  OSD  may  believe  that  there  have  been  a  sufficiently  large 
number  of  prior  experiences  in  the  area  of  live  fire  testing  to  indicate  that  pressure  by  one 
of  the  Services  for  successful  and  rapid  certification  of  its  products  under  development  can 
lead  to  inadequate  live  fire  testing  and  to  subsequent  unnecessary  combat  vulnerabilities. 
The  OSD,  therefore,  has  chosen  to  exercise  close  control  over  the  Live  Fire  Test  programs 
and  assumes  the  ultimate  authority  for  approval  of  the  equipment  based  on  the  program 
results. 

In  the  committee's  opinion,  the  Congress  may  believe  that  there  are  sufficient  proven 
instances  of  unnecessary  combat  vulnerability  in  DoD  equipment  previously  delivered  to  the 
field  to  warrant  legislative  direction  of  DoD  test  certification  to  include  live  fire  testing  of 
full-scale,  full-up  systems  using  munitions  likely  to  be  encountered  in  combat.  Congress 
further  believes  that  it  has  the  ultimate  responsibility  for  the  programs  it  authorizes  and 
therefore  has  the  obligation  to  exercise  that  legislative  direction. 

The  committee  believes  that  the  current  assessment  procedure,  which  is  supposed  to  result 
in  an  improved  aircraft,  does  not  guarantee  that  the  U.S.  armed  forces  will  field  cost-effective 
systems  designed  for  reduced  vulnerability.  The  intent  of  the  LFT  law  to  contribute  to  the 
creation  of  less  vulnerable  aircraft  designs  is  valid;  its  execution  to  achieve  this  intent  has 
been  flawed.  The  committee  believes  that  the  problems  with  the  LFT  law  are  (1)  ambiguities 
in  the  wording  of  the  law’s  requirements;  (2)  the  lack  of  a  clear  and  binding  LFT  policy 
directive;  (3)  the  Services'  reluctance  to  ask  for  a  waiver  from  full-scale,  full-up  LFT  for 
those  programs  for  which  they  believe  LFT  to  be  unreasonably  expensive  and  impractical 
because  of  the  fear  of  a  stigma  associated  with  the  waiver;  and  (4)  the  absence  of  a  formal 
waiver  process  that  includes  a  procedure  for  identifying  when  the  full-scale,  full-up  testing 
is  or  is  not  unreasonably  expensive  and  impractical,  which  would  eliminate  any  stigma 
associated  with  the  waiver. 


The  Future  of  Vulnerability  Assessment  The  committee  recognizes  the  limited  prospects 
for  both  new  program  starts  and  product  improvements  as  a  result  of  the  declining  DoD  budget. 
Although  this  new  environment  will  lead  to  austere  budgets  for  vulnerability  assessment,  the 
requirements  for  a  vulnerability  assessment  of  any  particular  system  should  not  decline. 
However,  the  overall  total  requirements  for  vulnerability  assessments  will  most  likely  decline 
due  to  the  reduction  in  the  number  of  active  aircraft  programs.  In  addition  to  individual 
program  cost  containment  issues,  the  committee  anticipates  a  reduction  in  both  the  analytical/ 
modeling  and  the  test  and  evaluation  infrastructure  within  OSD  and  each  of  the  Services  as 
the  total  DoD  budget  declines. 

Three  categories  for  cost  reduction  in  vulnerability  assessment  while  maintaining  or 
improving  the  current  capabilities  were  examined.  The  first  consists  of  an  increased  reliance 
on  analysis/modeling.  The  committee  believes  that  there  appears  to  be  a  sufficient  start  of 
a  modeling  capability,  and  of  a  weapons  effects  and  materials  data  base,  to  warrant  an 
increased  dependence  on  analysis/modeling  for  future  vulnerability  assessments  as  an  aid  in 
design.  However,  the  committee  also  believes  that  the  current  analytical  methodology  and 
supporting  data  bases  are  not  yet  sufficiently  robust,  correct,  precise,  and  representative  to 
permit  a  total  dependence  on  this  methodology.  Much  work  needs  to  be  accomplished  in 
model  development  and  in  the  accumulation  of  weapons  effects  and  material  vulnerability 
bases. 

A  reduction  in  assessment  costs  can  be  obtained  by  requesting  a  waiver  from  the  fu Li¬ 
se  ale  tests.  The  major  factor  in  the  cost  of  vulnerability  assessment  is  the  requirement  for 
the  full-scale  Live  Fire  Test  program  mandated  by  the  Live  Fire  Test  law.  The  law  offers  a 
waiver  from  the  full-scale,  full-up  tests  when  they  would  be  unreasonably  expensive  and 
impractical.  Under  the  current  LFT&E  Guidelines  and  Planning  Guide,  there  is  no  guidance 
as  to  what  constitutes  an  unreasonably  expensive  and  impractical  Live  Fire  Test  program. 
In  the  future,  a  procedure  must  be  established  for  gathering  the  facts  necessary  to  determine 
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if  the  full-scale,  full-up  Live  Fire  Tests  are  unreasonably  expensive  and  impractical  with 
respect  to  the  critical  vulnerability  issues  and,  if  they  are,  what  other  assessments  should  be 
conducted  in  place  of  the  complete  system  tests. 

A  third  possibility  for  cost  reduction  involves  a  consolidation  of  the  vulnerability  assess¬ 
ment  infrastructure.  Although  the  committee  did  not  review  in  detail  this  aspect  of  the 
vulnerability  assessment  activities  and  capabilities  in  each  of  the  three  Services,  the  committee 
believes  that  until  the  recent  DoD  budget  downturn,  there  were  sufficient  aircraft  programs 
to  warrant  the  continuation  of  more  or  less  similar  Service  live  fire  test  capabilities.  However, 
in  the  decades  ahead,  the  expected  requirements  for  the  vulnerability  testing  of  new  Service 
equipment  will  probably  fall  below  the  level  at  which  a  critical  mass  of  broad-based  facilities 
and  knowledgeable  staff  can  be  maintained  within  any  of  the  individual  Services.  Because 
considerable  cost  savings  could  be  achieved  by  consolidating  the  capabilities  of  these  facilities, 
some  form  of  consolidation  beyond  that  currently  contemplated  appears  inevitable. 

The  future  of  vulnerability  assessment  will  most  likely  involve  one  or  more  of  these  three 
categories. 


Conclusions 

After  reviewing  the  vulnerability  assessment  methodologies;  evaluating  the  cost,  effective¬ 
ness,  and  deficiencies  of  these  methodologies;  and  reviewing  and  evaluating  the  Live  Fire 
Test  law  and  the  OSD  and  Service  Live  Fire  Test  &  Evaluation  programs,  the  committee 
has  come  to  the  following  conclusions. 

•  Conclusions  Regarding  the  Live  Fire  Test  Law  <Sc  DoD  Programs 

1.  The  committee  believes  that  the  requirements  in  the  Live  Fire  Test  law  have 
been  interpreted  in  several  ways  and  that  these  different  interpretations  have  caused 
confusion  and  tension  in  the  Live  Fire  Test  programs.  Nevertheless,  the  committee 
believes  that  the  law  is  a  valuable  contribution  to  vulnerability  assessment  and  to  the 
design  of  survivable  aircraft  Furthermore,  it  is  satisfactory  in  its  present  form  because 
of  the  waiver  process.  The  committee  believes  that  the  law  has  had  a  positive  impact  on 
the  vulnerability  design  of  aircraft  and  is  sufficiently  flexible,  due  to  the  waiver  process,  to 
apply  to  all  aircraft  Furthermore,  the  committee  believes  that  verification  of  vulnerability 
by  live  fire  testing  is  necessary  and  that  this  law  ensures  that  verification. 

2.  The  committee  believes  that  the  1987  congressional  Live  Fire  Test  law  mandates 
live  fire  testing  of  full-scale,  full-up  aircraft,  including  on-board  ordnance,  unless  a 
waiver  is  granted  by  the  Secretary  of  Defense. 

3.  The  committee  believes  that  the  1988  Live  Fire  Test  &  Evaluation  Guidelines 
and  the  1989  Live  Fire  Test  &  Evaluation  Planning  Guide  are  not  consistent  with  its 
interpretation  of  the  LFT  law. 

4.  Because  all  three  Services  apparently  believe  that  an  LFT&E  program  plan  that 
contains  only  sub-scale  testing  is  in  compliance  with  the  law  as  interpreted  by  the  OSD 
1988  LFT&E  Guidelines,  no  current  LFT  program  contains  plans  to  conduct  full-scale 
tests  and  no  waivers  have  been  requested.  (The  committee  has  been  informed  that  on  May 
1 1, 1992,  the  Under  Secretary  certified  to  the  Congress  that  live  fire  testing  of  the  F/A-18E/ 
F  aircraft  would  be  unreasonably  expensive  and  impractical.  The  alternatives  to  the  statutorily 
prescribed  survivability  testing  are  being  prepared  by  the  Navy.) 

5.  The  committee  believes  that  a  waiver  is  required  to  omit  the  full-scale,  full-up 
tests. 

6.  The  committee  believes  that  there  are  aircraft  for  which  a  full-scale,  full-up  test 
program  is  unreasonably  expensive  and  impractical,  and  that  there  are  aircraft  for  which 
a  full-scale,  full-up  test  program  is  neither  unreasonably  expensive  nor  impracticaL 

7.  The  committee  believes  there  should  be  no  stigma  attached  to  a  waiver  because 
the  waiver  is  an  acceptable  alternative  LFT&E  path. 
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8.  A  serious  problem  in  both  the  analyses  and  the  Joint  Live  Fire  Testing  of  aircraft 
has  been  the  omission  of  on-board  ordnance  as  a  critical  component. 

9.  The  stated  intent  of  the  system  tests  mandated  by  the  Live  Fire  Test  law  is  to  aid 
in  design  by  providing  information  on  possible  weaknesses  sufficiently  early  in  the 
design  process  to  allow  the  weaknesses  to  be  corrected. 

10.  The  implied  intent  of  the  Live  Fire  Test  law  is  to  force  the  consideration  of 
vulnerability  during  the  design  process. 

11.  The  lack  of  a  definition  of  the  specific  threat  munitions  to  be  used  in  design  and 
in  Live  Fire  Testing  has  resulted  in  considerable  controversy  regarding  which  threat 
weapons  to  use  in  the  Services’  LFT  programs. 

12.  The  apparent  separation  of  the  oversight  of  vulnerability  analysis  from  the 
oversight  of  live  fire  testing,  both  of  which  are  part  of  the  testing  and  evaluation  process, 
has  created  a  situation  that  is  detrimental  to  the  overall  OSD  vulnerability  program. 

•  Conclusions  Regarding  the  Vulnerability  Assessment  Methodologies 

13.  Based  upon  its  review  of  the  two  methodologies,  the  committee  concludes  that 
both  vulnerability  analysis  and  live  fire  testing,  including  the  mandated  Live  Fire 
Testing,  are  essential  in  a  mix  peculiar  to  each  aircraft  development  program. 

14.  The  committee  believes  that  both  methodologies  need  to  be  improved  and  that 
these  improvements  should  be  mutually  beneficial. 

•  Conclusions  Regarding  the  Vulnerability  Programs  for  Aircraft 

15.  The  vulnerability  of  currently  fielded  U.S.  aircraft  will  become  more  important 
in  the  future. 

16.  There  is  insufficient  attention  given  to  the  requirement  to  design  for  vulnerability. 

17.  The  collection  of  actual  combat  data  on  the  vulnerability  of  U.S.  aircraft  is  not 
given  proper  emphasis. 

•  Conclusions  Regarding  the  Vulnerability  Infrastructure 

18.  The  process  of  designing  and  testing  for  vulnerability  is  extremely  complex  and 
would  benefit  from  continuous  input  and  oversight  from  a  broad  range  of  experts  in 
the  vulnerability  community. 

19.  The  vulnerability  community  of  the  future  most  likely  will  become  smaller  in 
both  the  number  of  programs  and  the  size  of  the  infrastructure. 

Recommendations 

Based  upon  the  results  of  the  committee’s  study  and  the  conclusions  given  above,  the 
committee  makes  the  following  recommendations: 

•  Recommendations  Regarding  the  DoD  Live  Fire  Test  &  Evaluation  Program 

1.  The  committee  recommends  that  the  Director,  Test  and  Evaluation,  issue  Guide¬ 
lines  that  replace  the  1988  Live  Fire  Test  &  Evaluation  Guidelines  and  that  more  clearly 
conform  with  the  requirements  for  the  full-scale,  full-up  tests  mandated  by  the  Live 
Fire  Test  law.  The  recommended  directive  should  completely  define  the  procedures  and 
requirements  for  planning  and  conducting  the  LFT&E  program  for  both  sub-scale  and  full- 
scale  tests.  The  directive  should  require  the  conduct  of  vulnerability  tests  under  realistic 
combat  conditions,  first  at  the  sub-scale  level  as  sub-scale  systems  are  developed,  and  later 
at  the  full-scale  level  mandated  in  the  legislation.  In  addition,  the  directive  should  describe 
a  formal  process  for  requesting  a  waiver. 

2.  The  committee  recommends  that  the  Director,  Test  and  Evaluation,  formalize  the 
waiver  process  by  developing  a  risk-benefit  assessment  methodology  that  can  be  used 
uniformly  to  determine  whether  a  full-scale,  full-up  test  program  for  any  particular 
aircraft  is  “unreasonably  expensive  and  impractical.”  The  methodology  must  also  be 
applicable  to  the  evaluation  of  the  alternative  Live  Fire  Test  program  for  the  sub-scale 
targets.  The  process  for  requesting  a  waiver,  described  in  the  DoD  directive  recommended 
above,  should  include  a  risk-benefit  assessment  methodology  that  quantifies  the  benefits 
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associated  with  full-scale,  full-up  Live  Fire  Tests  and  the  risks  associated  with  waiving  these 
tests.  Once  the  benefits  and  risks  have  been  quantified,  a  decision  can  be  made  as  to  whether 
the  full-scale,  full-up  tests  are  unreasonably  expensive  and  impractical.  The  committee 
strongly  believes  that  such  a  methodology  would  significantly  improve  the  process  of  re¬ 
questing  a  waiver. 

3.  The  committee  recommends  that  the  Secretary  of  Defense  take  measures  to  ensure 
that  (a)  the  LFT&E  Guidelines  are  properly  enforced  by  requiring  either  that  covered 
systems  be  subjected  to  full-scale,  full-up  testing  or  that  a  waiver  be  obtained;  (b)  that 
any  waiver  be  fully  justified;  (c)  that  the  waiver  process  be  uniformly  applied;  and  (d) 
that  no  stigma  be  attached  to  the  use  of  the  waiver  process. 

4.  The  committee  recommends,  for  the  full-scale,  full-up  Live  Fire  Tests,  that  the 
specific  ‘likely  to  be  encountered”  munitions  referred  to  in  the  Live  Fire  Test  law  be 
the  weapon(s)  specified  in  the  requirements  documentation  for  the  system,  projected 
forward  to  the  time  when  the  system  is  to  be  fielded.  Furthermore,  the  threat  should 
be  reviewed  and  updated  periodically  at  the  milestone  decision  points  to  ensure  that 
the  specified  design  weapon(s)  is  representative  of  the  major  “likely  to  be  encountered” 
threat(s)  to  the  system. 

5.  The  committee  recommends  that  the  Director,  Test  and  Evaluation,  expand  the 
charter  of  the  Live  Fire  Test  and  Evaluation  program  from  its  current  oversight  of 
those  tests  that  are  part  of  the  congressionally  mandated  Live  Fire  Test  program  to 
include  oversight  of  vulnerability  assessment 

•  Recommendation  Regarding  the  Vulnerability  Assessment  Methodologies 

6.  The  committee  recommends  that  both  the  analysis  community  and  the  live  fire 
testing  community  routinely  include  on-board  ordnance  in  their  assessments.  A  waiver 
to  allow  full-scale  Live  Fire  Tests  without  on-board  ordnance  should  be  granted  only  after 
an  examination  of  the  results  from  alternate  live  fire  tests  of  sub-scale  components  and  their 
integration  into  analyses  of  the  full-up  aircraft  carrying  such  ordnance. 

7.  The  committee  recommends  that  the  Under  Secretary  of  Defense,  Acquisition, 
direct  the  multi-Service  coordinated  development  and  authorization  for  use  of  improved 
analytical  vulnerability  assessment  models  that  are  applicable  to  all  military  aircraft 
The  current  Joint  Technical  Coordinating  Group  on  Aircraft  Survivability  (JTCG/AS)  ap¬ 
proved  models  could  form  the  basis  for  the  new  models.  The  1987  General  Accounting 
Office  study  on  Live  Fire  Testing  provides  many  suggestions  on  how  to  improve  these 
models. 

8.  The  committee  recommends  that  a  long-term  live  fire  test  program  be  funded  in 
which  realistic  components,  subsystems,  and  systems  are  specifically  tested  to  develop 
a  data  base  to  support  the  analytical  models. 

9.  The  committee  recommends  that  the  Secretary  of  Defense  (a)  establish  a  program 
to  examine  the  combat  data  collected  from  Desert  Storm  for  “lessons  learned”  regarding 
the  susceptibility  and  vulnerability  of  U.S.  and  allied  aircraft;  and  (b)  develop  formal, 
institutionalized  procedures  for  collecting  data  in  future  conflicts,  for  ensuring  that  the 
data  collectors  have  access  to  the  theater,  and  for  permanently  storing  the  data.  The 
combat  survivability  data  collection  program  should  reflect  the  importance  of  collecting  and 
preserving  the  data  and  should  be  coordinated  among  the  three  Services  through  a  joint 
agency,  such  as  the  JTCG/AS. 

•  Recommendations  Regarding  Vulnerability  Programs  for  Aircraft 

10.  Because  of  the  expected  service  life  extension  of  currently  fielded  U.S.  military 
aircraft,  the  committee  recommends  that  the  Under  Secretary  of  Defense,  Acquisition, 
establish  a  formal  vulnerability  assessment  and  reduction  program  for  these  aircraft 
This  program  should  require  that  all  product  improvement  or  upgrade  programs  to  existing 
aircraft  include  vulnerability  assessment  and,  if  appropriate,  reduction  as  major  goals  of  the 
program. 

11.  The  committee  recommends  (a)  that  a  vulnerability  assessment  program  be  an 
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integral  part  of  every  aircraft  acquisition  program;  (b)  that  vulnerability  assessment 
and  evaluation  be  a  specific  item  examined  at  each  formal  milestone  review;  and  (c) 
that  adequate  funds  be  appropriated  to  the  program. 

12.  The  committee  recommends  that  aircraft  programs  that  become  “prototype” 
programs,  such  as  the  RAH-66,  not  be  excluded  from  live  fire  testing.  The  committee  is 
concerned  that  the  RAH-66  might  be  developed  as  a  prototype  without  adequate  consideration 
or  testing  of  its  vulnerability.  If  the  decision  is  made  at  a  later  date  to  go  into  production 
with  the  prototype,  it  will  be  too  late  to  correct  any  design  weaknesses. 

13.  The  committee  recommends  that  specific  vulnerability  requirements  on  the  design 
be  a  part  of  the  survivability  objectives  defined  at  Milestones  I  and  II.  These  vulnerability 
requirements  should  be  identified  as  part  of  the  survivability  characteristics  and  incorporated 
in  the  aircraft  development  contracts. 

•  Recommendations  Regarding  the  Vulnerability  Infrastructure 

14.  The  committee  recommends  that  the  Director,  Test  and  Evaluation,  establish  a 
permanent  Senior  Vulnerability  Assessment  Board  comprised  of  senior  Services’  techni¬ 
cal  leaders,  high-level  OSD  officials,  and  nationally  recognized  experts  from  industry 
and  academia.  This  board  would  be  advisory  to  the  Director,  Test  and  Evaluation,  and 
chartered  to  review  annually  the  proposed  vulnerability  assessment  programs  and  budgets 
of  DoD  and  to  review  the  vulnerability  assessment  programs  on  specific  aircraft  as  the  need 
arises.  This  board  would  be  similar  to  the  boards  already  formed  for  conduct  of  coordinated 
6.1,  6.2,  and  6.3a  Tech  Base  programs  in  the  DoD. 

15.  The  committee  recommends  that  studies  be  conducted  to  determine  if  the  existing 
Army,  Navy,  and  Air  Force  vulnerability  analysis  community,  test  facilities,  and  infra¬ 
structure  can  be  reduced  proportionally  to  the  expected  overall  infrastructure  reduction 
within  DoD.  Project  Reliance,  the  existing  senior  joint  Services’  R&D  cooperation  group, 
should  be  charged  with  conducting  the  studies  of  how  best  to  accomplish  a  meaningful 
infrastructure  reduction.  The  committee  believes  that  this  drawdown  should  be  carried  out 
very  carefully  to  ensure  that  essential  vulnerability  assessment  personnel,  capabilities,  and 
facilities  are  not  lost  in  the  process. 

The  Future 

The  committee  recommends  to  the  Secretary  of  Defense  that  the  broad  issue  of  how 
to  both  design  and  test  for  vulnerability  in  an  austere  future  be  studied.  Present  concepts 
of  analyses  and  live  fire  testing  for  vulnerability  may  not  be  adequate  ip  a  future  of  reduced 
budgets,  fewer  fielded  aircraft,  fewer  program  starts,  smaller  procurement  numbers,  and  more 
“storage  on  the  shelf’  of  technology  capabilities  with  less  time  to  react  to  emergencies. 
When  such  a  study  has  been  completed  and  an  effective  process  has  been  developed  for 
vulnerability  design  and  validation,  OSD  should  consult  with  Congress  regarding  revisions 
to  the  LFT  law  that  reflect  this  new  process. 
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Review  of  Current 
Methodologies  Used  to  Assess 
Aircraft  Vulnerability  and 
Identification  of  Applications 
of  the  Results1 


What  Arc  the  Threats  to  Military  Aircraft? 

When  the  military  began  to  use  aircraft  in  war,  the  oppos¬ 
ing  forces  began  using  weapons  in  an  attempt  to  destroy 
them.  In  the  first  half  of  the  twentieth  century,  guns  were 
the  primary  weapons  used  against  aircraft.  These  guns  were 
either  surface-based  or  carried  by  enemy  aircraft  They 
ranged  from  the  small  arms  weapons,  such  as  the  0.3/0.303- 
inch  (7.62/7.7-millimeter)  and  0.50-caiiber  (12.7-millime- 
ter)  machine  guns,  to  anti-aircraft  artillery  (AAA),  such  as 
the  40-millimeter  and  88-millimeter  caliber  guns  of  World 
War  II  (WW II).  Contemporary  guns  that  can  be  used  against 
aircraft  include  the  5.56-millimeter,  7.62-millimeter,  12.7- 
millimeter,  14.5-millimeter,  and  20-millimeter  small  arms, 
and  the  23 -millimeter,  30-millimeter,  37-millimeter,  57-mil¬ 
limeter,  76-millimeter,  85-millimeter,  and  120-millimeter 
AAA.  The  small  arms  weapons  typically  fire  ball  ammuni¬ 
tion,  or  armor-piercing  projectiles,  known  as  AP  rounds,  or 
AP  projectiles  with  incendiaries,  known  as  API  rounds.  The 
AAA  weapons  and  the  larger-caliber  aircraft  guns  usually 
fire  ballistic  projectiles  with  a  high-explosive  (HE)  core 
and  a  surrounding  metal  case.  These  are  referred  to  as  HE 
warheads  or  HEI  warheads  when  incendiaries  are  included.2 


'Much  of  the  materiel  presented  in  this  chapter  is  based  upon  Ball 
(1985). 

:Some  of  the  small-caliber  AAA  also  fire  API  rounds. 


The  HE  warheads  may  detonate  on  contact  with  the  aircraft 
(contact-fuzed  HE  warheads),  after  an  elapsed  time  since 
firing  (time-fuzed  HE  warheads),  or  in  proximity  to  the 
aircraft  (proximity-fuzed  HE  warheads). 

After  World  War  H.  guided  missiles,  both  surface-based 
and  airborne,  were  developed  to  kill  aircraft.  These  anti¬ 
air  weapons  typically  cany  contact-  or  proximity-fuzed  HE 
warheads  designed  to  kill  aircraft  with  fragments  and  blast. 
Guns  and  guided  missiles  are  still  the  primary  threat  faced 
by  aircraft  today.  However,  several  new  threats  to  aircraft 
are  in  development  Directed  energy  weapons,  in  the  form 
of  low-to-medium  power  lasers  and  high-power  microwaves, 
have  the  potential  to  damage  or  destroy  sensors  on  the  air¬ 
craft  and  the  weapons  they  are  carrying;  and  high-power 
lasers  can  damage  major  aircraft  structure.  Chemical  and 
biological  weapons  pose  a  threat  to  aircraft,  particularly  on 
the  surface,  and  nuclear  weapons  are  a  threat  to  aircraft  on 
the  surface  and  in  the  air. 

What  Is  Aircraft  Vulnerability? 

Aircraft  survive  a  mission  into  hostile  territory  by 
“avoiding”  the  damage-causing  mechanisms  of  the  enemy’s 
air  defense  and  by  “withstanding”  the  damage  caused  by 
these  mechanisms  when  they  cannot  be  avoided.  The  aircraft 
attribute  known  as  susceptibility  refen  to  the  inability  of 
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the  aircraft  to  avoid  (being  damaged  by)  the  man-made 
hostile  environment  and  is  measured  by  PH,  the  probability 
the  aircraft  is  hit  by  a  weapon  while  on  its  mission.  The 
aircraft  attribute  known  as  vulnerability  refers  to  the  inability 
of  the  aircraft  to  withstand  (the  damage  caused  by  the)  hostile 
environment  and  is  measured  by  P**,  the  probability  the 
aircraft  is  killed3  given  that  it  is  hit.  The  probability  the 
aircraft  is  killed  by  a  particular  weapon  while  on  the  mission 
is  Pk,  which  is  equal  to  Ph#Pk*.  The  probability  the  aircraft 
survives  the  encounter  with  the  weapon  is  Ps,  which  is  equal 
to  l  -  PK,  which  is  the  same  as  1  -  Ph*SW  Thus,  reducing 
an  aircraft’s  susceptibility  (PH)  and  vulnerability  (P*^)  to  the 
weapons  likely  to  be  encountered  in  combat  increases  its 
survivability.  An  aircraft’s  susceptibility  can  be  reduced  by 
destroying  the  enemy  air  defense  elements,  by  reducing  the 
aircraft’s  signatures  (stealth),  by  employing  on-board  and 
off-board  threat  warning  systems  and  electronic  countermea¬ 
sures,  and  by  the  tactics  employed.  An  aircraft’s  vulnerabil¬ 
ity  can  be  reduced  by  using  redundant  and  separated  compo¬ 
nents,  by  locating  components  to  minimize  the  possibility 
and  extent  of  damage,  by  designing  components  to  contain 
or  withstand  the  effects  of  damage,  by  adding  special  equip¬ 
ment  to  suppress  the  damage,  by  shielding  components,  and 
by  removing  vulnerable  components  from  the  design.  A  very 
important  aspect  of  vulnerability  reduction  is  that  many 
design  features  are  effective  against  a  number  of  different 
threat  weapons.  For  example,  locating  redundant  flight  con¬ 
trol  hydraulic  components  on  opposite  sides  of  the  aircraft 
and  inerting  the  fuel  tank  ullages  will  provide  protection 
from  both  gun  projectiles  and  proximity-fuzed  missiles  in 
most  situations.  Thus,  in  many  situations  it  is  not  necessary 
to  consider  all  of  the  individual  threats  when  designing  the 
aircraft. 


Critical  Components  and  Essential  Functions .  Each  com¬ 
ponent  in  the  aircraft  has  a  level,  degree,  or  amount  of 
vulnerability  to  the  damage-causing  mechanisms4  generated 
by  the  threat  weapon;  and  each  component’s  vulnerability 
contributes  in  some  measure  to  the  vulnerability  of  the  total 


3The  word  kill  is  used  here  in  «  general  sense.  The  vulnerability  assess¬ 
ment  community  uses  several  definitions  of  kill.  Two  categories  of  kill  are 
the  attrition  kill  and  the  mission  abort  kill.  There  are  several  levels  of 
attrition  kill  based  upon  the  elapsed  tune  of  kill  after  the  hit.  For  example, 
the  K-level  attrition  kill  is  defined  as  a  kill  in  which  the  aircraft  falls  out 
of  control  within  30  seconds  after  the  hit.  and  the  A  level  is  defined  as  a 
kill  in  which  the  aircraft  falls  out  of  control  within  5  minutes  after  the  hit 
'Damage,  threat,  or  kill  mechanisms  are  the  output  of  the  threat  warhead 
that  cause  damage  to  the  aircraft.  The  types  of  damage  mechanisms  associ¬ 
ated  with  penetrator  and  high-explosive  warheads  are  penetraiors,  frag¬ 
ments.  incendiaries,  and  blast.  Damage  processes  refer  to  the  interaction 
of  the  damage  mechanism  with  the  aircraft  and  its  components.  The  damage 
processes  associated  with  the  damage  mechanisms  listed  here  include  ballis¬ 
tic  impact,  penetration,  combustion  (in  the  form  of  a  fire  or  explosion), 
hydraulic  or  hydrodynamic  ram,  and  blast  loading. 


aircraft.  The  critical  components  on  an  aircraft  are  those 
components  whose  kill  result  in  the  loss  of  an  essential 
function.  Essential  functions  are  those  functions  required  to 
prevent  an  aircraft  kill.  The  essential  functions  that  prevent 
an  attrition  kill  are  lift,  thrust,  and  control  of  flight,  and  the 
ability  to  land  safely.  Navigation  and  weapons  delivery  are 
two  possible  essential  functions  for  a  mission  abort  kill.  An 
example  of  a  critical  component  for  the  attrition  kill  is  the 
single  pilot  who  controls  the  flight  of  the  aircraft.  If  the 
pilot  is  killed  (i.e.,  he/she  is  unable  to  perform  the  essential 
function  of  control  of  the  aircraft)  the  aircraft  is  also  killed. 
An  example  of  a  critical  component  on  an  attack  aircraft 
for  the  mission  abort  kill  is  the  weapons  delivery  computer. 
If  the  computer  is  killed,  the  weapons  cannot  be  released  at 
the  correct  time;  consequently,  the  pilot  will  return  to  base 
prior  to  mission  completion. 

Components  that  do  not  contribute  to  any  of  the  essential 
functions  become  critical  when  their  response  to  a  hit  (i.e., 
their  kill  mode)  causes  the  kill  of  another  component  that 
is  critical  because  it  contributes  to  an  essential  function.  For 
example,  consider  the  bombs  carried  on-board  an  attack 
aircraft.  The  bombs  do  not  contribute  to  the  essential  func¬ 
tions  for  flight  of  lift,  thrust,  and  control.  However,  if  one 
of  the  bombs  explodes  when  hit  by  a  fragment  or  bullet, 
and  the  explosion  kills  the  pilot  or  any  other  critical  compo¬ 
nents  on  the  aircraft,  the  bombs  are  critical  components 
because  their  kill  mode  (explosion)  eventually  leads  to  a  kill 
of  the  aircraft.3  The  propagation  of  damage  from  the  hit 
component  to  other  components  is  known  as  cascading  dam¬ 
age.  Pyrotechnic  items,  such  as  infrared  flares,  are  also  criti¬ 
cal  components  when  their  reaction  to  a  hit  leads  to  a  fire 
and  the  eventual  loss  of  the  aircraft. 

The  critical  components  can  be  nonredundant,  such  as 
the  single  pilot  and  single  engine  on  a  single-piloted,  single- 
engined  aircraft,  or  redundant,  such  as  the  two  engines  on 
a  two-engined  aircraft  When  the  critical  components  are 
redundant,  a  kill  of  more  than  one  of  the  redundant  compo¬ 
nents  is  required  for  a  kill  of  the  aircraft  In  general,  the 
critical  components  on  a  particular  aircraft  depend  only  upon 
the  selected  kill  category  (and  level,  if  appropriate)  and  the 
assumed  kill  mode(s),  and  not  upon  the  threat  weapon.6 

The  procedure  used  to  determine  ail  of  the  nonredundant 
and  redundant  critical  components  on  an  aircraft  is  known 
as  the  critical  component  analysis.  Two  different  types  of 
analyses  can  be  used,  the  Failure  Mode  and  Effects  Analysis 
(FMEA)  and  the  Fault  Tree  Analysis  (FTA).  In  the  FMEA, 
all  possible  failure,  damage,  or  kill  modes  of  a  component 
or  subsystem  are  identified  and  the  consequence  of  each 


5The  treatment  of  the  on-board  munitions  when  assessing  aircraft  vulner¬ 
ability  is  a  major  concern  to  the  committee,  particularly  for  aircraft  with 
internal  ordnance  storage.  This  concern  is  examined  in  detail  in  Chapters 
2  and  4. 

‘Refer  to  footnote  3  for  several  examples  of  kill  definitions. 


REVIEW  OF  CURRENT  METHODOLOGIES  .AND  IDENTIFICATION  OF  .APPLICATIONS 


13 


TABLE  l-l  List  of  Some  Subsystem  Damage-Caused  Failure  (Kill)  Modes  [Ball,  1985) 

Fuel  Subsystem 

Propulsion  Subsystem 

Flight  Control  Subsystem 

Fuel  supply  depletion 

Fuel  ingestion 

Disruption  of  control  path 

In-tank  fire/explosion 

Foreign  object  ingestion 

Loss  of  control  power 

Void  space  fire/explosion 

Inlet  flow  distortion 

Loss  of  aircraft  motion  data 

Sustained  exterior  fire 

Lubrication  starvation 

Damage  to  control  surfaces 

Hydraulic  ram 

Compressor  case  perforation 
Combustor  case  perforation 

Hydraulic  fluid  fire 

Power  TrainlRotor 

Turbine  section  failure 

Structural  Subsystem 

Blade! Propellor  Subsystem 

Structural  removal 

Loss  of  lubrication 

Exhaust  duct  failure 

Pressure  overload 

Mechanicai/structural  damage 

Engine  control/accessories 

failure 

Thermal  weakening 

Electrical  Subsystem 

Penetration 

Severing  or  grounding 

Crew  Subsystem 

Mechanical  failure 

Injury,  incapacitation,  or  death 

Avionics  Subsystem 

Overheating 

Penetrator/fragment  damage 

Armament  Subsystem 
Fire/explosion 

Fire/explosion/overheat 

component  failure/damage/kill  mode  upon  each  of  the  essen¬ 
tial  functions  is  determined.7  In  the  FT  A,  those  component 
or  subsystem  kill  modes  required  to  cause  the  loss  of  the 
essential  functions  are  determined. 

Kill  Modes.  For  many  years,  the  aircraft  vulnerability  com¬ 
munity  has  observed  the  results  of  live  fire  testing  of  compo¬ 
nents,  subsystems,  and  aircraft  and  has  examined  the  combat 
data  on  damaged  and  killed  aircraft  in  order  to  determine 
all  of  the  kill  modes  associated  with  each  of  the  aircraft 
subsystems.  For  example,  there  are  five  kill  modes  associated 
with  the  fuel  subsystem.  When  a  fuel  tank  is  holed  by  a 
penetrator  or  fragment,  a  catastrophic  explosion  or  major 
fire  may  occur  inside  the  tank,  or  fuel  may  leak  from  the 
hole  in  the  tank  into  an  adjacent  void  space  or  dry  bay  and 
catch  fire,  or  hydraulic  ram  damage  to  the  fuel  tank  wall 
may  cause  a  major  structural  failure  of  the  tank  or  allow 
fuel  to  dump  into  engine  intake  ducts,  causing  an  engine 
kill.  A  list  of  some  of  the  possible  kill  modes  for  each  of 
the  major  subsystems  on  an  aircraft  has  been  compiled  based 
upon  these  observations  and  studies.  This  list  is  presented 
in  Table  1-1. 

The  kill  modes  listed  in  Table  1-1  describe  different  types 
of  reaction  that  components  or  subsystems  in  the  aircraft 
exhibit  when  the  aircraft  is  hit  In  some  of  the  kill  modes, 
the  component  hit  is  the  only  component  killed,  whereas  in 
others,  the  component  hit  reacts  to  the  hit  in  a  mode  that 


The  relation  between  a  component  or  system  failure  mode  and  combat- 
caused  damage  or  kill  modes  is  developed  in  the  Damage  Mode  and  Effects 
Analysis. 


kills  other  components.  An  example  of  the  former  is  the 
loss  of  flight  control  due  to  a  hit  in  a  hydraulic  power  actuator 
that  causes  a  jam  of  the  actuator  and  a  loss  of  control  of  the 
control  surface.  An  example  of  the  latter  is  a  fuel  ingestion 
kill  of  an  engine  due  to  a  hit  on  a  fuel  tank  adjacent  to  the 
air  inlet  Reducing  the  vulnerability  of  an  aircraft  to  the 
threat  weapons  and  their  damage  mechanisms  involves  re¬ 
ducing  the  likelihood  the  kill  modes  given  in  Table  1-1  will 
occur  when  the  aircraft  is  hit 

The  Failure  Mode  and  Effects  Analysis  (FMEA).  As  an 
example  of  the  FMEA  process,  consider  a  single-engine 
aircraft  with  only  two  fuel  tanks,  one  in  each  wing.  The 
tanks  are  partially  full,  and  there  are  fuel  vapors  in  the  ullage4 
of  the  tanks.  The  possible  kill  modes  for  the  fuel  subsystem 
are  given  in  Table  1-1.  One  fuel  tank  kill  mode  is  an  explo¬ 
sion  inside  the  tank.  If  the  consequence  of  the  internal  explo¬ 
sion  in  either  wing  tank  is  the  destruction  of  the  wing  con¬ 
taining  the  tank,  which  then  causes  a  kill  of  the  aircraft  due 
to  loss  of  lift,  both  wing  fuel  tanks  are  nonredundant  critical 
components  for  the  attrition  kill  for  the  internal  explosion 
kill  mode.  On  the  other  hand,  suppose  the  kill  mode  of  the 
tanks  is  a  loss  of  fuel  storage  capability  due  to  one  or  more 
holes  in  the  bottom  of  the  tank.  If  this  kill  mode  occurs  in 
only  one  tank,  this  will  not  lead  to  a  loss  of  thrust  due  to 
fuel  supply  depletion  when  the  undamaged  tank  can  provide 
fuel  to  the  engine.  However,  if  both  tanks  are  holed  and 
lose  their  storage  capability,  then  a  fuel  supply  depletion 


The  ullage  is  the  volume  of  the  tank  above  the  fuel  level.  Fuel  v apart 
accumulate  in  the  ullage. 
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kill  will  occur,  the  aircraft  will  lose  thrust,  and  an  attrition 
kill  will  result.  Thus,  for  this  kill  mode,  the  fuel  tanks  are 
redundant  critical  components. 

The  Fault  Tree  Analysis  (FTA).  In  the  FTA  process,  the 
selected  kill  category  (and  possibly  level)  is  defined  as  the 
top-level  undesirable  event,  and  the  component  kill  required 
to  cause  the  undesirable  event  are  determined.  The  compo¬ 
nent  kill  that  result  in  the  undesired  event  are  linked  together 
in  the  fault  tree  by  using  logical  AND  and  OR  gates.  For 
example,  consider  an  aircraft  with  components  A,  B,  C,  and 
D.  An  undesirable  kill  will  occur  if  either  component  A  OR 
B  is  killed,  or  it  may  occur  if  both  components  C  AND  D 
are  killed.  Thus,  components  A  and  B  are  nonredundant 
critical  components,  and  components  C  and  D  are  redundant 
critical  components.  In  using  FTA  for  the  fuel  tank  example 
given  above,  one  undesirable  event  leading  to  an  attrition 
kill  is  loss  of  lift.  If  loss  of  lift  occurs  due  to  an  explosion 
inside  the  left  wing  fuel  tank,  a  component  A  kill,  OR  if  it 
occurs  due  to  an  explosion  inside  the  right  wing  tank,  a 
component  B  kill,  both  wing  fuel  tanks  are  nonredundant 
critical  components  for  the  explosion  kill  mode.  On  the  other 
hand,  a  loss  of  thrust  will  occur  if  wing  tanks  A  AND  B 
are  killed  (by  the  fuel  supply  depletion  kill  mode).  Thus, 
the  tanks  are  redundant  critical  components  for  this  kill 
mode.  As  another  example  of  FTA,  consider  a  two-engined 
aircraft  The  undesired  event  of  loss  of  thrust,  which  leads 
to  an  attrition  kill,  will  occur  when  the  left  engine  AND  the 
right  engine  are  killed.  Thus,  these  two  components  are 
redundant  critical  components.  A  list  of  the  typical  critical 
components  on  a  single-piloted,  two-engined  helicopter  is 
given  in  Table  1-2. 

The  Kill  Tree.  A  visual  illustration  of  all  of  the  critical 


components  and  their  redundancies  is  provided  by  the  kill 
tree,9  such  as  the  one  shown  in  Figure  l-l  for  an  attrition 
kill  of  a  two-engined,,  two-piloted  helicopter.  A  complete 
horizontal  or  diagonal  cut  through  the  tree  trunk  anywhere 
along  the  trunk  will  cause  a  kill.  For  example,  a  kill  of  the 
pilot  and  either  the  copilot  or  the  copilot’s  controls  will 
cause  a  kill,  as  will  a  kill  of  the  drive  train  or  any  of  the 
three  cyclic  actuators.  If  the  kill  mode  of  the  left-  and  right- 
hand  fuel  tanks  is  fuel  supply  depletion,  both  tanks  must  be 
killed  to  cause  a  kill  of  the  aircraft.  On  the  other  hand,  if 
the  kill  mode  is  a  fuel  fire  or  explosion,  then  a  kill  of  either 
tank  will  kill  the  aircraft.  Once  the  critical  components  have 
been  identified  and  arranged  in  the  kill  tree,  a  vulnerability 
assessment  can  be  performed. 

What  Is  a  Vulnerability  Assessment? 

A  vulnerability  assessment  is  broadly  defined  here  as 
the  systematic  description,  delineation,  test  and  evaluation, 
analysis,  or  quantification  of  the  vulnerability  of  the  individ¬ 
ual  critical  components  and  of  the  total  aircraft.  When  an 
aircraft  is  hit  by  one  or  more  damage  mechanisms  generated 
by  the  threat  weapon,  the  outcome  of  those  hits  is  not  deter¬ 
ministic;  it  is  random  or  stochastic.10  For  example,  when  15 
fragments  from  a  proximity-fuzed  high-explosive  warhead 
penetrate  the  upper  wall  of  an  aircraft’s  wing  fuel  tank,  the 
flammable  vapor  inside  the  tank  may  explode,  destroying 
the  wing  and  killing  the  aircraft;  or  the  vapor  may  not 


The  kill  tree  is  also  referred  to  as  the  fault  tree. 

I0A  deterministic  process  has  a  repeatable  outcome  that  can  be  predicted 
with  certainty  if  all  of  the  influencing  parameters  and  governing  laws  are 
known.  Random  or  stochastic  processes  have  multipie  or  various  outcomes, 
any  one  of  which  may  or  may  not  occur  on  any  one  trial. 


TABLE  1-2  List  of  Typical  Nonredundant  and  Redundant  Critical  Components  on  a 
Single-Piloted,  Two-Engined  Helicopter  (Ball,  1985) 

Nonredundant  Critical  Components  Redundant  Critical  Components 


Flight  Control  Subsystem  Components 
Rods,  bellcranks,  pitch  links,  swashplate, 
hydraulic  actuators,  collective  lever,  and 
control  pedals 

Rotor  Blade  and  Rower  Train  Components 
Blades,  drive  shafts,  rotor  heads,  main 
transmission,  and  gearboxes 

Fuel  Subsystem  Components 
Fuel  cells,  sump,  lines,  and  valves 

Structural  Subsystem  Components 
Tail  boom 


Propulsion  Subsystem  Components 
Engines  and  engine  mounts 

Hydraulic  Subsystem  Components 
Hydraulic  reservoirs,  lines,  and  components 

Structural  Subsystem  Components 
Redundant  structural  elements 
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FIGURE  M  The  attrition  kill  tree  for  a  two- piloted,  two-engined  helicopter  (Ball,  1985).  Copyright  ©  AIAA  1985 — Used  with  permission. 


explode,  and  the  aircraft  survives  the  15  hits.  The  likelihood 
of  an  explosion  inside  the  tank  depends  upon  many  random 
variables,  such  as  the  amount  of  fuel  vapor,  the  oxygen 
concentration  in  the  vicinity  of  the  fragments,  and  the  tem¬ 
perature  of  the  fragments. 


How  Is  Vulnerability  Measured? 

As  a  consequence  of  the  random  nature  of  vulnerability, 
the  metric  most  often  used  to  quantify  the  vulnerability  of 
an  aircraft’s  critical  components  is  P*,  the  probability  the 
component  is  killed  given  a  random  hit  on  the  component 
by  a  threat  weapon  or  damage  mechanism.11  The  value  of 
Pw,  depends  upon  the  intensity  of  the  terminal  effects  parame¬ 
ters  associated  with  the  damage  mechanism,  such  as  mass 
and  impact  velocity  on  the  component  for  penetrators  and 
fragments.  The  set  of  component  Pw  values  for  different 
masses  and  impact  velocities  is  known  as  the  Pw  function. 
A  second  metric  used  to  quantify  a  component’s  vulnerabil¬ 
ity  is  A,,  the  vulnerable  area  of  the  component  Component 
vulnerable  area  is  defined  as  the  presented  area  of  the  compo¬ 
nent  that  if  hit  would  cause  a  kill  of  the  component  and  is 
equal  to  the  product  of  the  component’s  presented  area  A* 
in  the  threat  approach  direction  and  its  P^,,  i.e..  A,  =  A^P**. 


“Other  metrics  sometimes  used  for  component  vulnerability  are  P**, 
the  probability  a  component  is  damaged  given  a  hit,  area  removal,  energy 
density,  and  blast 


The  metrics  used  to  quantify  the  vulnerability  of  the 
aircraft  to  a  single  random  hit  by  a  penetrator  or  contact- 
fuzed  warhead  include  P^,  the  probability  the  aircraft  is 
killed  given  a  random  hit  on  the  aircraft  and  Av,  the  aircraft’s 
single  hit  vulnerable  area.12  The  metric  used  to  quantify  the 
vulnerability  of  an  aircraft  to  the  proximity-  and  time-fuzed 
HE  warheads  on  AAA  projectiles  and  guided  missiles  is  Pjot,, 
the  probability  the  aircraft  is  killed  given  an  external  detona¬ 
tion  by  a  high-explosive  warhead.  The  P, ^  is  a  function 
of  the  location  of  the  detonation  point  with  respect  to  the 
aircraft. 


What  Are  the  Two  Methodologies  Used  to  Assess 
Vulnerability? 

In  general,  there  are  two  methodologies  used  to  assess 
aircraft  vulnerability.  One  method  is  the  a  priori  prediction 
of  aircraft  vulnerability  by  using  analyses  or  modeling.  This 
method  is  nearly  always  supported  by  prior  live  fire  test 
data  on  component  P^  values  for  the  various  kill  modes. 
However,  the  data  have  often  been  obtained  on  older  equip¬ 
ment.  The  other  method  is  the  a  posteriori  observation  and 


“Lowercase  subscripts  refer  to  a  component  and  uppercase  subscripts 
refer  to  the  aircraft  Thus,  It*  is  the  probability  a  component  is  killed  given 
a  random  hit  on  the  component,  is  the  probability  a  component  is  killed 
given  a  random  hit  on  the  aircraft  and  is  the  probability  the  aircraft 
is  killed  given  a  random  hit  on  the  aircraft 
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possible  measurement  of  aircraft  vulnerability  by  using  em¬ 
pirical  data  obtained  from  either  actual  combat,  aircraft  acci¬ 
dents,  or  controlled  live  fire  testing.13  This  method  is  nearly 
always  supported  by  a  priori  predictions  of  vulnerability 
prior  to  testing  to  define  the  test  conditions  and  by  a  posteriori 
analyses  or  evaluation  of  the  data,  A  brief  review  of  the 
state-of-the-art  of  vulnerability  analysis/modeling  and  vul¬ 
nerability  testing  is  given  below. 


Analysis! Modeling..  The  prediction  of  an  aircraft’s  vulner¬ 
ability  to  the  ballistic  projectiles  and  guided  missiles  likely 
to  be  encountered  in  combat  can  be  accomplished  by  using 
standardized  computer  programs.14  One  set  of  programs  is 
applicable  to  a  single  hit  by  impacting  penetrator  or  frag¬ 
ment.  Computation  of  Vulnerable  Area  and  Repair  Time 
(CO V ART)  is  the  Joint  Technical  Coordinating  Group  on 
Aircraft  Survivability  (JTCG/AS)  standard  program  for  com¬ 
puting  the  critical  component  vulnerable  areas  A*  and  the 
aircraft’s  vulnerable  area  Av  for  a  single  random  hit  by  a 
penetrator  or  fragment  (JTCG/ME,  1984).  Another  set  of 
programs  computes  aircraft  vulnerability  to  contact-fuzed 
HE  warheads  that  detonate  on  the  surface  or  within  the 
aircraft.  High  Explosive  Vulnerable  Area  and  Repair  Time 
(HEVART)  (BRL,  1978  and  HEI  Vulnerability  Assessment 
Model  (HEIVAM)  (Datatec  Inc.,  1979)  are  examples  of  this 
type  of  program.  A  third  set,  known  as  endgame  programs, 
computes  the  probability  an  aircraft  is  killed  due  to  an  exter¬ 
nal  burst  of  an  HE  warhead.  SCAN  (Dayton  University  Ohio 
Research  Institute,  1976)  is  the  current  JTCG/AS  endgame 
model  for  computing  an  aircraft’s  P^D.  Modular  Endgame 
Computer  Assessment  (MECA),  Joint  Services  Endgame 
Model  (JSEM),  SESTEM  U  (ASD/WPAFB,  1981),  and 
SHAZAM  (Air  Force  Armament  Lab./Eglin  AFB,  1983)  are 
four  other  widely  used  endgame  programs. 

All  of  these  vulnerability  assessment  programs  require 
as  input  a  three-dimensional  data  base  that  defines  the  geo¬ 
metric  model  of  the  aircraft.  The  geometric  model  may  be 
contained  within  the  vulnerability  assessment  program,  as 
in  SCAN,  or  it  may  be  developed  in  a  separate  program, 
such  as  MAGIC,  Ballistic  Research  Laboratory  Computer- 
Aided  Design  (BRL-CAD)  package,  or  FASTGEN  III, 
which  are  used  as  preprocessors  for  COVART.  This  model 
should  contain  all  of  the  aircraft’s  components,  equipment, 
and  supplies,  including  such  items  as  fuel,  hydraulic  fluid. 


“Combat  and  accident  data  are  extremely  valuable  as  adjuncts  to  the 
other  methodologies,  but  they  are  limited  in  scope,  limited  in  the  information 
on  the  nature  of  the  event,  and  not  always  available  for  direct  application. 

,4The  Joint  Technical  Coordinating  Group  on  Airciaft  Survivability  has 
established  a  library  of  computer  programs  for  assessing  the  susceptibility, 
vulnerability,  and  survivability  of  aircraft  The  library  is  maintained  arid 
operated  by  the  Survivability/Vulnerability  Information  and  Analysis  Cen¬ 
ter  (SURVTAO  at  the  Wright  Aeronautical  Laboratories. 


and  ordnance.  However,  because  of  the  limitations  on  pro¬ 
gram  size,  available  time,  and  manpower,  many  small  non- 
critical  components  that  are  not  expected  to  influence  the 
results  are  often  omitted.13  Another  subsystem  that  has  often 
been  omitted  in  vulnerability  assessments  is  the  on-board 
ordnance  in  the  form  of  bombs,  missile  warheads  and  propel¬ 
lants,  and  ammunition  drums.  On  most  aircraft,  bombs  and 
missiles  are  carried  externally.  In  this  position,  they  may 
shield  other  components  from  projectiles  and  fragments,  or 
they  may  react  violently  to  a  ballistic  impact  (e.g.,  detonate) 
and  destroy  the  aircraft.  The  new  stealth  aircraft  carry  ord¬ 
nance  internally  in  order  to  reduce  signatures.  Adverse  reac¬ 
tions  of  any  internally  carried  ordnance,  such  as  a  deflagra¬ 
tion  or  a  detonation,  have  an  even  greater  probability  of 
destroying  the  aircraft.  The  omission  of  on-board  ordnance 
from  the  assessment  is  discussed  in  more  detail  in  Chapters 
2  and  4. 

Another  input  requirement  for  the  assessment  is  the  kill 
tree  (or  logical  kill  expression)  for  the  selected  kill  category 
(and  level  if  appropriate).  This  tree  defines  the  redundant 
and  nonredundant  components  that  if  killed  individually  (the 
single  engine  on  a  single-engined  aircraft)  or  in  combination 
(both  engines  on  a  two-engined  aircraft)  will  cause  an  aircraft 
kill.  Associated  with  each  critical  component  on  the  tree  is 
a  data  base  that  contains  the  or  A,  value  for  the  component 
that  is  based  upon  the  selected  threat  weapon  or  damage 
mechanism  and  the  possible  range  of  impact  velocities  on 
the  installed  component,  for  the  kill  modes  considered  in 
the  critical  component  analysis. 

Vulnerability  to  a  Single  Hit  by  a  Penetrator  or  Frag¬ 
ment.  All  of  the  vulnerability  assessment  programs  contain 
an  assumption  as  to  how  the  damage  mechanisms  associated 
with  the  weapon  proceed  through  the  aircraft.  The  COVART 
methodology  assumes  that  the  penetrator  or  fragment  from 
any  selected  direction16  is  equally  likely  to  impact  the  aircraft 
at  any  location  and  that  it  propagates  along  a  straight  line, 
known  as  a  shotline,  through  the  aircraft,  slowing  down  and 
possibly  breaking  up  as  it  penetrates  the  various  components. 
The  amount  of  fragment  or  penetrator  slowdown  is  deter¬ 
mined  by  the  penetration  equations  that  are  a  part  of  the 
built-in  data  base.  Ricochet  of  the  fragment  or  penetrator  is 
not  considered.  An  additional  assumption  often  made  is  that 
only  the  components  that  are  intersected  by  one  shotiine  can 
be  killed  by  the  hit  along  that  shotline.  This  assumption 
rules  out  the  possibility  of  cascading  damage  away  from  the 
shotline.17  In  the  analysis,  the  presented  area  of  the  aircraft 


“The  COVART  model  for  the  F-22  contains  2.213  components,  of 
which  nearly  half  are  critical. 

'*The  directions  usually  selected  include  the  six  cardinal  views  of  front, 
back,  top,  bottom,  left  side,  and  right  side,  and  may  include  the  twenty 
45-degree  angles  between  these  six  views. 

I7lt  is  possible  to  modify  the  intersected  component’s  P*  to  account 
for  kills  of  adjacent  components. 
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FIGURE  1-2  Example  of  a  grid  and  random  shotlines  from  FASTGEN  for  COVART  (Ball,  1985).  Copyright  ©  AIAA  1985— Used 
with  permission. 


from  the  selected  direction  is  covered  by  a  uniform  grid, 
and  one  shotline  is  randomly  located  within  each  cell.  An 
example  of  the  random  shotlines  within  the  cells  for  a  partic¬ 
ular  aircraft  is  shown  in  Figure  1-2. 

The  user  has  the  option  of  selecting  the  uniform  cell  size. 
Typical  cell  sizes  range  from  12  inches  to  1  inch  on  a  side, 
with  2  inches  being  typical.  A  preprocessor  program,  known 
as  a  shotline  generator  program,  such  as  MAGIC,  BRL- 
CAD,  or  FASTGEN  III,  identifies  all  of  the  critical  compo¬ 
nents  intersected  by  each  shotiine.  This  information  is  input 
data  for  COVART.  COVART  computes  the  vulnerable  area 
of  each  critical  component  and  the  aircraft’s  single  hit  vul¬ 
nerable  area,  as  well  as  the  probability  the  aircraft  is  killed 
by  a  random  hit  For  component  vulnerable  areas,  each  grid 
cell  containing  a  shotline  that  intersects  a  component  has  a 
vulnerable  area  equal  to  the  product  of  the  presented  area 
of  the  cell  and  the  P^,  for  the  shotline  through  the  component 
The  total  vulnerable  area  of  the  component  is  the  sum  of 
the  vulnerable  areas  of  those  cells  with  shotlines  that  inter¬ 
sect  the  component.  For  the  aircraft  vulnerable  area  Ay,  each 
grid  cell  shown  in  Figure  1-2  contributes  a  vulnerable  area 
equal  to  the  product  of  the  presented  area  of  the  cell  and 
the  probability  the  aircraft  is  killed  by  a  hit  along  the  shotline 
in  that  cell.1*  The  total  aircraft  vulnerable  area  is  equal 
to  the  sum  of  the  vulnerable  areas  of  each  of  the  cells. 
Consequently,  redundant  components,  if  separated,  that  both 
are  not  intersected  by  one  shotline,  do  not  contribute  to  the 
aircraft’s  single  hit  vulnerable  area  for  that  shotline.19  The 
Puh  for  the  aircraft  is  equal  to  the  Av  of  the  aircraft  divided  by 
Ap,  the  aircraft’s  presented  area  from  the  selected  direction. 


"When  more  chan  one  nonrcdundant  critical  component  is  intersected 
by  a  shotiine.  the  probability  the  aiacraft  is  killed  is  equal  to  the  union  of 
the  component  probabilities  of  kill. 

"This  is  the  result  of  the  assumption  that  only  those  components  inter¬ 
sected  by  the  shotiine  can  be  killed.  A  modification  of  the  P»  value  for 
a  component  can  be  made  to  allow  a  hit  on  one  component  to  cause  a  kill 
of  another  component  due  to  cascading  damage. 


Vulnerability  to  a  Contact-Fuzed  High-Explosive  War¬ 
head.  Essentially  the  same  analytical  procedure  is  followed 
for  contact-fuzed  high-explosive  warheads.  A  geometric 
model  of  the  aircraft,  the  kill  tree,  and  the  critical  component 
Pm,  or  Av  data  are  required.  A  grid  is  superimposed  on  the 
aircraft  and  a  shotiine  is  randomly  located  within  each  cell. 
The  difference  between  this  analysis  for  the  contact-fuzed 
HE  warhead  and  the  analysis  for  the  single  penetrator  or 
fragment  is  the  fact  that  components  in  the  vicinity  of  the 
shotiine  can  be  killed  by  the  blast  and  fragments  from  the 
detonation  of  the  HE  warhead.  Thus,  redundant  critical  com¬ 
ponents  that  are  relatively  close  together  can  be  killed  by  a 
single  hit,  causing  a  kill  of  the  aircraft.  Figure  1-3  shows 
the  grid  cell  and  randomly  located  shotlines  for  this  type  of 
analysis.  Note  that  in  this  figure  the  HE  warhead  detonation 
can  cause  a  kill  of  both  the  fuel  tank  and  the  engine  even 
though  neither  component  was  hit  directly  by  the  weapon. 

Vulnerability  to  an  Externally  Detonating  High-Explosive 
Warhead.  The  analysis  for  the  externally  detonating  HE 
warhead,  shown  in  Figure  1-4,  follows  the  same  procedure 
used  for  the  single  penetrator  or  fragment,  except  that  the 
fragment  shotlines  emanating  from  the  external  detonation 
are  radial  rather  than  parallel,  and  the  aircraft  can  suffer 
multiple  fragment  impacts  over  its  surface  rather  than  a 
single  hit.  In  addition,  the  blast  from  the  detonation  can  kill 
the  aircraft  The  assessment  of  the  kill  of  the  aircraft  by 
external  blast  is  usually  made  independently  from  the  frag¬ 
ment  assessment.  Three-dimensional  blast  contours  around 
the  aircraft  are  determined  as  a  function  of  HE  weight. 
Within  a  particular  blast  kill  contour  for  particular  explosive 
charge  weight  a  detonation  of  a  warhead  with  that  charge 
weight  or  larger  will  kill  the  aircraft 

Results  from  the  Analyses.  The  results  or  information  ob¬ 
tained  from  an  analytical  assessment  of  aircraft  vulnerability 
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FIGURE  1*3  Grid  cells  and  shotlines  for  the  contact-fuzed  high  explosive  weapon  (Ball,  1985).  Copyright  <©  AIAA  1985 — Used  with 
permission. 


FIGURE  1-4  •  Aircraft  vulnerability  to  the  externally  detonating  HE  warhead  (Ball.  1985).  Copyright  ®  AIAA  1985 — Used  with  permission. 

for  the  single  hit  by  a  penetrator  or  fragment  typically  con-  by  the  contact-fuzed  high-explosive  warhead  consist  of  the 

sists  of  predictions  of  the  values  of  vulnerable  area  A,  for  aircraft  vulnerable  area  A?  and  the  probability  of  kill  given 

all  of  the  critical  components,  the  aircraft  vulnerable  area  a  random  hit  on  the  aircraft  P^.  The  results  of  an  assessment 

Av,  the  probability  the  aircraft  is  killed  given  a  hit  within  for  the  externally  detonating  warhead  consist  of  the  probabtl* 

each  grid  cell,  and  the  probability  the  aircraft  is  killed  given  ity  of  kill  of  the  critical  components  intersected  by  the  frag* 

a  random  hit  ?m.  The  assessment  results  for  the  single  hit  ment  shotlines  from  the  warhead  detonation,  the  probability 
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of  aircraft  kill  due  to  blast,  and  the  probability  of  aircraft 
kill  given  a  detonation 

The  Stochastic  Qualitative  Analysis  of  System  Hierarchies 
(SQuASH)  Model .  One  of  the  primary  criticisms  of  the 
current  aircraft  vulnerability  models  is  the  straight  shotline 
assumption.  Fragments  and  penetrators  usually  do  not  pene¬ 
trate  through  the  aircraft  in  a  straight  line.  In  an  attempt  to 
account  for  the  random,  irregular  path  of  penetrators  and 
fragments  through  armored  vehicles.  Ballistic  Research  Lab¬ 
oratory  (BRL)  developed  the  SQuASH  model  (Deitz  et  al., 
1990).  SQuASH  is  applicable  to  both  penetrator  and  high- 
explosive  weapons.  It  allows  for  deflections  of  the  pene¬ 
trators  and  fragments  from  the  straight  shotline,  the  creation 
of  spall,  and  it  tracks  the  pieces  of  fractured  penetrators. 
The  present  version  of  SQuASH  was  developed  for  the 
vulnerability  analysis  of  armored  ground  vehicles.  However, 
its  methodology  could  be  applied  to  aircraft 

The  model  introduces  the  concept  of  Spaces.  All  possible 
warhead  and  target  conditions  at  the  time  of  the  hit  form 
the  Initial  Conditions  Space,  or  Space  1 .  A  particular  set  of 
conditions,  such  as  the  type  and  operational  status  of  the 
target  and  the  location  of  the  hit  on  the  target,  is  one  point 
in  the  Initial  Conditions  Space.  Due  to  the  hit,  some  compo¬ 
nents  will  be  damaged,  and  some  will  be  killed.  These  dam¬ 
aged  and  killed  components,  and  all  other  post-event  observ¬ 
ables,  such  as  holes  in  plates  and  other  terminal  effects, 
form  the  damage  vector.  All  possible  damage  vectors  for 
the  target  form  the  Damage  Space,  or  Space  2,  and  the 
specific  damage  vector  containing  the  components  damaged 
or  killed  by  the  hit  is  one  point  in  the  Damage  Space.  All 
possible  target  capabilities  after  the  hit  form  the  Capabilities 
Space,  or  Space  4,  and  the  particular  target  capabilities  re¬ 
maining  after  the  hit  represent  one  point  in  the  Capabilities 
Space.20  The  vulnerability  event  starts  with  a  point  in  the 
Initial  Conditions  Space.  This  point  is  mapped  to  the  Damage 
Space  either  by  a  live  fire  test  or  by  the  SQuASH  model. 
Note  that  because  the  vulnerability  event  is  nondeterminisdc, 
one  point  in  the  Initial  Conditions  Space  can  map  to  many 
different  points  in  the  Damage  Space.  The  mapping  from 
the  Damage  Space  to  the  Target  Capabilities  Space  is  accom¬ 
plished  currently  by  using  the  Damage  Assessment  List  In 
the  future,  the  Degraded  States  methodology  will  be  used 
for  this  mapping. 

SQuASH  is  a  Monte  Carlo  model.  Each  shot  at  the  target 
is  replicated  (typically  1,000  times)  with  slight  variations  in 
its  initial  conditions.  For  each  replication  or  trial,  random 
drawings  determine  which  events  (such  as  kill  of  a  compo¬ 
nent  that  is  hit)  occur.  The  resulting  damage  vector  for  that 


xSpace  3  represents  objective  Measures  of  Performance  and  is  not 
modeled. 
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shot  is  computed,  and  the  frequency  of  occurrence  of  the  ele¬ 
ments  in  these  damage  vectors  is  produced  as  an  intermediate 
result.  Input  kill  or  fault  trees  are  used  to  develop  estimates 
of  target  and  individual  subsystem  kill  probabilities. 

Some  difficulties  associated  with  SQuASH  are  the  lack 
of  data  with  respect  to  the  broken  paths  and  component 
damage,  especially  synergistic  damage,  and  the  problems 
associated  with  relating  component  damage  to  degradation 
in  performance.  Another  difficulty  is  the  magnitude  of  the 
number  of  possible  outcomes  from  one  event.  This  number 
is  dependent  upon  the  number  of  components  that  can  be 
killed.  There  may  be  a  large  number  of  components  to  con¬ 
sider  for  a  particular  shot;  perhaps  between  10  and  100.  The 
number  of  components  in  an  entire  aircraft  might  be  on  the 
order  of  1,000.  The  damage  vector  consists  of  these  M 
components,  and  each  of  the  M  components  or  elements  in 
the  vector  is  either  a  0  (no  damage)  or  a  l  (damage),  and 
the  sample  space  is  said  to  have  dimension  M.  The  sample 
space  of  possible  combinations  of  components  that  might 
be  damaged  by  a  particular  shot  is  2M.  Thus,  the  sample 
space  for  a  given  shot  can  be  quite  large.  Some  sort  of  metric 
is  needed  to  reduce  the  sample  space  to  one  with  a  more 
manageable  size.  One  approach  might  be  to  create  some  sort 
of  metric  that  quantifies  the  “nearness”  of  various  damage 
vectors  (similar  to  a  Hamming  distance). 


T e  sting.  As  a  result  of  the  random  nature  of  the  vulnerabil¬ 
ity  problem,  the  multitude  of  known  component  or  subsys¬ 
tem  kill  modes,  the  possible  existence  of  unknown  or  previ¬ 
ously  unobserved  kill  modes  or  cascading  damage,  and  the 
difficulty  in  quantifying  the  vulnerability  of  the  components 
and  subsystems  for  each  of  these  kill  modes,  the  use  of 
combat  data21  and  the  results  from  controlled  live  fire22  tests 
have  always  been  integral  parts  of  vulnerability  assessment. 
These  data  provide  insight  into  the  component  and  subsys¬ 
tem  kill  modes  and  any  cascading  damage  that  can  occur. 
Furthermore,  when  a  sufficiently  targe  number  of  identical 
tests  are  performed,  statistical  data  on  vulnerabilityare  gen¬ 
erated.  However,  because  of  the  expenses  and  difficulty 
associated  with  obtaining  large  quantities  of  useful  results 
from  either  combat  or  testing,  there  is  a  general  reluctance 
to  engage  in  large-scale  efforts  that  may  provide  little  useful 
data  or  may  have  little  applicability  to  present  or  future 
aircraft  or  analytical  models.  Nevertheless,  many  live  fire 
tests  have  been  conducted  since  WW  U  on  targets  ranging 


!>The  combat  data  gathered  in  past  conflicts  is  stored  in  the  Combat 
Data  Information  Center,  which  is  part  of  the  Survivability /V ulnerability 
Information  and  Analysis  Center.  SURVIAC  is  located  at  Wright- Patterson 
Air  Force  Base,  Ohio. 

^The  texm  “live  fire  testing”  is  used  here  in  the  general  sense  to  mean 
firing  live  (both  explosive  and  non-expiosive)  ammunition  or  fragments  at 
the  target  (and  hitting  it). 
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TABLE  1-3  Definitions  of  Types  of  Test  Articles  and  Type  of  Tests  (GAO,  1987) 

Loading 

Scale 

Full-Up 

Inert 

Full-scale 
(Complete  System ) 

Complete  system  with 
combustibles 

Complete  system  without 
combustibles 

Sub-scale 
( Partial  System) 

Components,  subsystems,  or 
subassemblies  with 
combustibles 

Components,  subsystems,  or 
subassemblies  without 
combustibles 

from  individual  components  to  actual  aircraft.  Of  particular 
interest  here  are  the  current  Joint  Live  Fire  (JLF)  program 
and  the  congressionally  mandated  Live  Fire  Test  and  Evalua¬ 
tion  (LFT&E)  program. 

General  Procedure  for  Testing.  Before  reviewing  the  JLF 
and  LFT  programs,  the  general  procedure  for  testing  that 
has  been  established  by  the  vulnerability  testing  community 
is  described.  Briefly,  one  or  more  targets  and  weapons  are 
obtained  and  prepared  for  testing.  The  target  can  be  one 
component,  a  subassembly,  a  subsystem,  several  subsys¬ 
tems,  portions  of  the  aircraft,  or  the  aircraft  weapon  system. 
According  to  the  General  Accounting  Office  (GAO),  tests 
conducted  on  the  complete  weapon  system  are  known  as 
full-scale  tests,  and  tests  on  less  than  full-scale  targets  are 
known  as  sub-scale  tests.  Corresponding  definitions  also 
used  in  this  report  are  complete  system  tests  and  partial 
system  tests.  A  surrogate  target  or  weapon  is  an  existing 
target  or  weapon  that  is  similar  to  the  intended  target  or 
weapon.  If  the  target,  either  the  complete  system  or  a  partial 
system,  contains  all  of  the  appropriate  combustibles,  such  as 
fuel,  hydraulic  fluid,  ordnance,  and  stowage  items  normally 
found  on  the  aircraft  when  operating  in  combat,  the  tests  are 
known  as  full-up  tests.  Inert  targets  lack  all  of  the  appropriate 
combustibles,  and  semi-inert  targets  contain  some  of  the 
combustibles.  Table  1-3  contains  these  definitions,  which 
are  used  throughout  this  report 

The  Test  Plan  and  Some  Important  Considerations.  The 
test  plan  contains  the  test  objectives  and  the  issues  the  tests 
are  supposed  to  provide  information  on,  the  weapon  to  be 
used,  the  selection  and  placement  of  test  instrumentation, 
the  selection  of  the  number  of  shots,  the  shotline  directions, 
the  impact  locations,  and  any  analytical  methods  that  will 
be  used.  The  test  plan  may  contain  a  number  of  random 
shots  as  well  as  a  number  of  selected  shots.  The  tests  are 
scheduled  so  those  shots  that  are  expected  to  cause  minimum 
damage  to  the  target  are  conducted  early  in  the  program. 
Those  tests  that  are  expected  to  cause  more  severe  damage 


are  conducted  at  the  end  of  the  program.  Particular  shots 
that  have  the  potential  to  destroy  the  target,  although  of  vital 
interest,  may  not  be  conducted  at  all.  Preparation  for  testing 
consists  of  the  preparation  of  the  test  site,  the  weapon,  and 
the  target.  After  each  test,  the  target  is  repaired  and  returned 
to  a  condition  as  similar  to  the  original  condition  as  possible. 
If  the  weapon  is  a  non-explosive  penetrator  or  fragment,  the 
amount  of  damage  is  usually  small,  the  repairs  are  relatively 
simple,  and  the  target  can  be  hit  in  essentially  the  same 
location  again.  However,  if  the  weapon  contains  a  high- 
explosive  warhead,  the  damage  is  more  severe  and  extensive, 
the  repair  is  more  difficult,  and  it  may  not  be  possible  to 
return  the  aircraft  to  its  original  condition.  In  this  situation, 
the  shotline  for  a  second  shot  must  be  sufficiently  separated 
from  the  first  shotline  so  that  the  damage  and  subsequent 
repair  of  the  first  shot  do  not  influence  the  results  of  the 
second  shot. 

Some  of  the  important  test  considerations  are  the  external 
and  internal  environmental  conditions  at  the  time  of  the  test, 
such  as  the  requirements  for  external  air  flow  over  the  target, 
and  the  proper  fuel  vapor  states  and  temperatures  inside  the 
target;  the  requirement  for  jig  arrangements  to  introduce 
loads  on  the  aircraft  structure;  and  the  requirement  for  all 
of  the  equipment  to  be  operating  at  the  time  of  the  hit.  For 
example,  must  a  helicopter  rotor  blade  or  tail  rotor  drive 
shaft  be  turning  when  it  is  hit  by  the  weapon?  Must  the 
hydraulic  fluid  be  at  the  normal  operating  temperature  when 
the  line  is  hit?  What  internal  structural  loads  are  appropriate 
for  the  test,  those  associated  with  normal  flight,  or  those 
associated  with  a  violently  maneuvering  aircraft? 

The  Test  Results .  The  results  or  information  obtained  from 
controlled  live  fire  tests  typically  consists  of  a  list  of  the 
components  that  were  damaged  or  killed,  the  nature  and 
severity  of  the  damage,  the  kill  modes  observed  and  any 
cascading  damage,  and  an  estimate  or  measurement  of  the 
ability  of  the  aircraft  to  continue  the  operation  of  essential 
functions.  Specific  events,  such  as  the  initiation  of  a  fire 
and  the  intensity  and  duration  of  the  fire,  are  also  noted. 
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Due  to  the  randomness  of  the  reactions  to  the  hit,  some  of 
the  observed  results  in  one  test  may  not  be  observed  in  any 
of  the  other  tests.  For  example,  firing  a  12. 7-millimeter  API 
into  a  partially  empty  fuel  tank  may  not  result  in  an  internal 
explosion  on  the  first  test  shot,  but  the  second  shot  may 
cause  an  explosion  that  destroys  the  tank.  On  the  first  firing 
of  a  12.7-millimeter  API  projectile  into  a  helicopter  engine 
nacelle,  the  projectile  may  ricochet  into  the  cockpit;  on  the 
second  shot,  it  may  ricochet  into  the  transmission. 

What  Are  the  Joint  Live  Fire  and  Live  Fire  Test 
Programs? 

Joint  Live  Fire .  In  1983,  the  Office,  Secretary  of  Defense 
(OSD)  Director,  Defense  Testing  and  Evaluation  nominated 
to  the  Services  a  joint  test  and  evaluation  initiative  for  the  live 
fire  of  munitions,  foreign  and  U.S.,  made  against  currently 
operational  full-scale  targets,  both  U.S.  and  foreign.  This 
program  is  known  as  the  Joint  Live  Fire  program.  The  U.S. 
targets  originally  included  land,  sea,  and  air,  however,  the 
sea  targets  were  eventually  excluded  from  the  program.  Can¬ 
didate  aircraft  included  the  F-15,  F-16,  F/A-18,  AV-8B, 
fixed  wing  aircraft,  and  the  UH-60  and  AH-64  helicopters. 
The  threats  initially  considered  consisted  of  armor-piercing 
projectiles  with  incendiaries  (12.7-millimeter,  14.5-millime¬ 
ter,  23-millimeter,  and  30-millimeter  API),  warhead  frag¬ 
ments  (45, 70,  1 10,  and  220  grains),  and  contact-fuzed  high- 
explosive  rounds  with  incendiaries  (23-millimeter  and 
30-millimeter  HEI).  A  number  of  specific  tests  on  various 
components  and  subsystems  of  these  aircraft  have  been  con¬ 
ducted,  such  as  tests  on  the  UH-60  main  rotor  blade,  the 
F-15  and  F-16  hydraulic  fluid,  the  F-15  and  F-16  steady 
state  and  quick  dump  fuel  ingestion,  and  the  F-16  emergency 
power  subsystem.  In  1989,  the  results  of  the  JLF  tests  were 
presented  to  more  than  100  industry,  government,  and  mili¬ 
tary  specialists  in  vulnerability  and  vulnerability  testing.  The 
JLF  program  is  still  active.  The  test  data  gathered  during 
the  tests  are  currently  being  examined  to  determine  the  P^ 
values  for  the  tested  components,  and  the  empirical  values 
are  being  compared  to  the  previous  values  in  order  to  decide 
whether  the  previous  values  should  be  revised. 

The  Live  Fire  Test  Law .  As  a  result  of  the  controversy 
over  the  vulnerability  testing  of  the  U.S.  Army’s  Bradley 
Fighting  Vehicle,  Congress  in  fiscal  year  1987  amended  title 
10  of  the  U.S.  Code,  adding  Section  2366.  “Major  Systems 
and  Munitions  Programs:  Survivability  and  Lethality  Test¬ 
ing;  Operational  Testing.”'  This  legislation,  known  as  the 
Live  Fire  Test  (LFT)  law,  applies  to  covered  systems.  Ac¬ 
cording  to  the  law,  "A  covered  system  means  a  vehicle , 
weapon  platform ,  or  conventional  weapon  system  that  (A) 


includes  features  designed  to  provide  some  degree  of  protec - 
tion  to  users  in  combat ;  and  (B)  that  is  a  major  system 
within  the  meaning  of  that  term  is  section  2303(5)  of  title 
10"  (U.S.  Congress,  1986-1989).  A  major  system  is  one 
that  costs  $75  million  in  Research  Development,  Testing, 
and  Evaluation  and/or  $300  million  in  procurement,  in  1980 
dollars,  and  is  determined  by  the  Secretary  of  Defense  not 
to  be  aTiighly  classified  (i.e.,  black)  program.  Several  modi¬ 
fications  have  been  made  to  the  law  since  FY1987.23 

According  to  the  LFT  Test  Guidelines  established  by 
the  law,  “ Survivability  and  lethality  tests  required  under 
subsection  (a)  shall  be  carried  out  sufficiently  early  in  the 
development  phase  of  the  system  or  program  to  allow  any 
design  deficiency  demonstrated  by  the  testing  to  be  corrected 
in  the  design  of  the  system ,  munition,  or  missile  before 
proceeding  beyond  low-rate  initial  production "  Note  that 
survivability  is  used  when  vulnerability  is  intended.  The 
primary  requirement  of  the  law  is  that  "a  covered  system 
may  not  proceed  beyond  low-rate  initial  production  until 
realistic  survivability  testing  of  the  system  is  completed ' 
Realistic  survivability  testing  is  defined  as  “ testing  for  vul¬ 
nerability  of  the  system  in  combat  by  firing  munitions  likely 
to  be  encountered  in  combat  (or  munitions  with  a  capability 
similar  to  such  munitions)  at  the  system  configured  for  com¬ 
bat,  with  the  primary  emphasis  on  testing  vulnerability  with 
respect  to  potential  user  casualties  and  taking  into  equal 
consideration  the  susceptibility  to  attack  and  combat  perfor¬ 
mance  of  the  system "  "The  term  configured  for  combat, 
with  respect  to  a  weapon  system,  platform,  or  vehicle,  means 
loaded  or  equipped  with  all  dangerous  materials  (including 
allflammables  and  explosives )  that  would  normally  be  car¬ 
ried  in  combat" 

A  waiver  from  the  law  is  provided.  "The  Secretary  of 
Defense  may  waive  the  application  of  the  survivability  and 
lethality  tests  of  this  section  to  a  covered  system,  if  the 
Secretary,  before  the  system  or  program  enters  full-scale 
engineering  development ,  certifies  to  Congress  that  live-fire 
testing  of  such  system  or  program  would  be  unreasonably 
expensive  and  impractical "  Also,  "the  Secretary  shall  in¬ 
clude  with  any  such  certification  a  report  explaining  how 
the  Secretary  plans  to  evaluate  the  survivability  or  the  lethal¬ 
ity  of  the  system  or  program  and  assessing  possible  alterna¬ 
tives  to  realistic  survivability  testing  of  the  system  or  pro¬ 
gram"  (U.S.  Congress,  1986-1989). 

The  intent  of  the  LFT  law  is  to  determine  the  inherent 
strengths  and  weaknesses  of  adversary,  U.S.,  and  allied 
weapon  systems  sufficiently  early  in  the  program  to  allow 
any  design  deficiency  to  be  corrected.  According  to  the 
FY1988-1989  DoD  Authorization  Act  Conference  Report, 
Congress  intended  that  the  Secretary  of  Defense  implement 


aThe  law  and  the  amendments  to  the  law  are  Included  in  this  report 
in  Appendix  A 
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the  LFT  law  “in  a  manner  which  encourages  the  conduct  of 
full-up  vulnerability  and  lethality  tests  under  realistic  combat 
conditions,  first  at  the  sub-scale  level  as  they  are  developed, 
and  later  at  the  full-scale  level  mandated  in  the  legislation” 
(U.S.  Congress,  1988).  All  live  fire  tests  conducted  as  pan 
of  the  program  to  satisfy  the  Live  Fire  Test  law  will  be 
referred  to  here  as  Live  Fire  Tests.  Developmental  tests 
using  live  fire  that  are  not  intended  to  be  part  of  the  mandated 
LFT&E  program  will  be  referred  to  as  live  fire  tests,  with 
no  capital  letters.  The  distinction  between  the  two  categories 
of  tests  is  important. 

In  response  to  the  law,  the  Department  of  Defense  (DoD) 
chartered  an  administering  office,  the  Director  of  Live  Fire 
Testing,  under  the  Office  of  the  Director  of  Defense  Research 
and  Engineering.  The  responsibilities  of  this  office  include 
the  establishment  of  policies  under  which  Live  Fire  Testing 
is  conducted  by  the  Service  components,  the  approval  of 
the  Services'  Live  Fire  Test  strategy  and  test  plans  for  each 
covered  program,  the  review  of  the  test  results,  and  the 
performance  of  an  independent  assessment  that  is  forwarded, 
via  the  Secretary  of  Defense,  to  the  Congress  (O'Bryon, 
1991). 

What  Does  the  Law  Require?  During  the  course  of  the 
committee’s  examination  of  the  current  direction  of  Live 
Fire  Testing  and  Evaluation,  it  became  apparent  that  because 
of  the  ambiguity  of  the  law’s  requirements  regarding  the 
system  testing,  there  were  different  interpretations  of  the 
LFT  law.  One  interpretation  was  that  the  law  did  not  explic¬ 
itly  stipulate  that  a  complete  system  had  to  be  tested,  even 
though  no  waiver  from  the  law  was  requested.  The  opinion 
was  held  that  the  law  was  satisfied  by  an  LFT&E  program 
in  which  Live  Fire  Testing  was  conducted  only  on  compo¬ 
nents  and  subsystems,  provided  that  these  tests  showed  that 
no  complete  system  testing  was  necessary;  all  vulnerabilities 
had  been  found  in  the  partial  system  tests.  In  an  attempt  to 
determine  the  intent  of  Congress  as  to  the  meaning  of  realis¬ 
tic  survivability  testing,  members  of  the  committee  inter¬ 
viewed  Mr.  Joseph  Cirincione,  the  congressional  staff  mem¬ 
ber  who  drafted  the  Live  Fire  Test  legislation  in  1987.  Mr. 
Cirincione  believes  that  the  intent  of  the  law,  as  seen  by  the 
Congress,  is  “full-scale,  full-up”  testing.  This,  to  him,  means 
that  the  complete  aircraft  must  be  tested  and  must  be  config¬ 
ured  for  combat  (i.e.,  engine  running,  fuel  in  the  tanks, 
loaded  with  ammunition,  etc.).  He  believes  that  anything 
other  than  full-scale,  full-up  testing  requires  a  waiver  in 
accord  with  the  terms  of  the  above  paragraph.  In  support 
of  his  position  is  the  statement  in  the  FY1988-1989  DoD 
Authorization  Act  Conference  report  that  says  “the  conferees 
intend  that  the  Secretary  of  Defense  implement  this  section 
in  a  manner  which  encourages  the  conduct  of  full-up  vulnera¬ 
bility  and  lethality  tests  under  realistic  combat  conditions, 
first  at  the  sub-scale  level  as  sub-scale  systems  are  developed. 


and  later  at  the  full-scale  level  mandated  in  the  legislation” 
(U.S.  Congress,  1988).  Furthermore,  the  events  that  led  to 
the  law  and  the  fact  that  Congress  included  a  waiver  process 
in  the  law  are  further  evidence  that  Congress  intended  that 
live  fire  tests  be  conducted  on  full-scale,  full-up  systems.  If 
tests  on  the  full-scale,  full-up  system  were  not  intended,  no 
waiver  would  be  necessary,  and  any  live  fire  tests  would 
suffice,  as  long  as  they  were  realistic.  Based  upon  the  evi¬ 
dence  gathered  by  the  committee  and  its  study  of  the  law, 
the  committee  is  unanimous  in  the  opinion  that  the  LFT  law 
requires  a  full-scale,  full-up  aircraft  to  be  tested,  regardless 
of  the  outcome  of  the  sub-scale  tests,  unless  a  waiver  is 
granted. 


What  Are  the  Applications  of  the  Results  of  the 
Assessments? 

Vulnerability  assessments  are  a  part  of  the  weapon  system 
acquisition  process.  This  process  is  described  in  DoD  In¬ 
struction  (DODI)  5000.2,  February  23,  1991.  According  to 
DoDI  5000.2,  survivability  is  identified  as  a  critical  system 
characteristic  and  consequently  must  be  addressed  in  cost- 
schedule-performance  trade-offs  throughout  the  acquisition 
process.  This  instruction  requires  that  survivability  be  con¬ 
sidered  from  all  threats  found  in  the  various  levels  of  conflict, 
including  the  conventional  gun  and  missile  threats,  the  nu¬ 
clear,  biological,  and  chemical  threats,  and  the  advanced 
directed  energy  weapons.  At  Milestone  0,  the  expected  threat 
environment  is  identified  and  discussed  in  the  Mission  Need 
Statement.  At  Milestone  I,  the  system  threat  assessment 
identifies  the  expected  likelihood  for  each  threat.  In  addition, 
initial  survivability  objectives  are  defined  and  validation 
criteria  established  in  the  Operational  Requirements  Docu¬ 
ment  (ORD).  Key  objectives  are  included  in  the  Concept 
Baseline.  Critical  survivability  characteristics  and  issues  that 
require  test  and  evaluation  are  identified  and  included  in  the 
Test  and  Evaluation  Master  Plan;  this  includes  the  Live  Fire 
Test  program.  Critical  survivability  technology  shortfalls  are 
identified  and  research  requirements  established.  At  Mile¬ 
stone  II,  survivability  issues  are  addressed  in  the  Integrated 
Program  Summary;  at  Milestone  HI,  an  assessment  of  how 
well  the  survivability  objectives  have  been  met  has  been 
completed,  and  all  survivability  issues  should  have  been 
resolved. 

Vulnerability  objectives  are  part  of  the  survivability  ob¬ 
jectives  required  by  DoDI  5000.2.24  If  any  vulnerability  ob¬ 
jectives  or  requirements  have  been  defined  in  the  ORD,  they 
are  satisfied  and  validated  by  using  vulnerability  assessments 


:4Note  that  only  survivability  objectives  are  required.  Thus,  a  system 
could  meet  the  requirements  in  DoD  5000.2  by  requirements  on  susceptibil¬ 
ity  alone. 
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in  the  form  of  analysis/modeiing  and  testing.  Thus,  the  pri¬ 
mary  application  of  a  vulnerability  assessment  in  the  weapon 
system  acquisition  process  is  to  aid  in  the  design  of  the 
aircraft  and  in  the  validation  of  the  design.  Additional  appli¬ 
cations  are  to  satisfy  program  requirements,  to  develop  data 
bases  in  support  of  subsequent  analytical  assessments,  to 
predict  test  outcomes,  to  satisfy  the  requirements  of  the  Live 
Fire  Test  law,  and  to  support  acquisition  decisions. 

Aid  in  Design  and  Design  Validation .  The  results  of  a 
vulnerability  assessment  must  be  available  early  in  the  devel¬ 
opment  cycle  of  an  aircraft  and  used  to  influence  the  design. 
Analytical  modeling  can  provide  guidance  on  the  placement 
of  the  critical  components  and  the  protection  that  should  be 
given  to  the  various  contributors  to  vulnerability,  such  as 
the  fuel  subsystem,  flight  control  subsystem,  and  propulsion 
subsystem.  Controlled  live  fire  developmental  tests  can  be 
conducted  on  early  designs  of  components,  and  possibly 
subsystems,  to  determine  any  adverse  reactions,  either  ex¬ 
pected  or  unexpected.  Any  design  vulnerabilities  revealed 
by  the  full-scale,  full-up  LFTs  should  also  impact  the  design. 
For  design  validation,  the  analytical  models  provide  informa¬ 
tion  on  vulnerable  area,  P^,  and  P*^;  and  live  fire  tests  are 
conducted  to  verify  that  certain  vulnerability  requirements 
for  the  design  of  the  aircraft  have  been  satisfied.  For  example, 
if  an  aircraft  has  a  design  requirement  to  be  able  to  take  a 
single  hit  by  a  12.7-millimeter  API  anywhere  on  the  aircraft 
and  fly  for  30  minutes  after  the  hit,  live  fire  testing  of  the 
design  is  the  best  procedure  for  verifying  the  compliance  of 
the  design.23 

Satisfy  Program  Requirements .  The  DoD  MIL-STD 
2069,  “Requirements  for  Aircraft  Nonnuclear  Survivability 
Program,**  requires  that  analytical  vulnerability  assessments 
be  made  as  part  of  the  normal  development  process.  Aircraft 
development  programs  that  stipulate  MIL-STD-2069  will 
have  assessments  conducted  throughout  the  development 
cycle. 

Development  of  Data  Bases  in  Support  of  Subsequent  Ana¬ 
lytical  Assessments .  As  data  from  live  fire  tests  on  a  variety 
of  components  and  subsystems  are  gathered,  qualitative  in¬ 
formation  on  kill  modes  and  cascading  damage  effects  and 
quantitative  information  on  individual  component  func¬ 
tions  can  be  put  into  a  data  base  and  used  to  improve  subse¬ 
quent  analytical  assessments.  The  JLF  program  is  an  exam¬ 
ple  of  this  application  in  action.  Another  application  of  this 


I5Thc  design  requirement  thar  an  aircraft  be  able  to  withstand  a  single 
hit  by  a  particular  weapon  and  continue  to  fly  for  a  specified  period  of 
dme  does  not  automatically  mean  that  the  aircraft  will  be  unable  to  withstand 
a  second  hit.  Building  into  the  aircraft  an  ability  to  take  a  single  hit  anywhere 
also  gives  the  aircraft  a  significant  capability  to  withstand  multiple  hits. 


type  is  the  use  of  the  results  from  the  analytical  vulnerability 
assessments  in  trade-off  and  campaign  or  similar  large-scale 
war  game  models  that  require  an  aircraft  attrition  data  base. 

Predict  Test  Outcomes .  The  results  of  an  analytical  vulner¬ 
ability  assessment  can  be  used  to  predict  the  possible  out¬ 
comes  of  a  controlled  test  prior  to  the  conduct  of  the  test. 
The  particular  components  that  will  be  damaged  or  killed 
by  the  weapon  or  by  any  cascading  effects  can  be  identified, 
and  the  consequences  of  the  damage  or  kill  of  these  compo¬ 
nents  to  the  essential  functions  can  be  predicted.  However, 
due  to  the  random  nature  of  vulnerability,  no  deterministic 
prediction  of  the  test  outcome  can  be  made.  Consequently, 
predictions  take  the  form  of  statements  such  as  “the  flam¬ 
mable  vapors  in  the  wing  tank  have  a  0.3  probability  of 
exploding  and  destroying  the  wing  when  the  tank  is  hit  by 
a  12.7-millimeter  API.’* 

Satisfy  the  Requirements  of  the  Live  Fire  Test  Law .  The 
Live  Fire  Test  law  requires  that  a  full-scale  (the  complete 
weapon  system),  full-up  (configured  for  combat)  aircraft 
be  tested  for  vulnerability  using  munitions  likely  to  be 
encountered  in  combat,  unless  a  waiver  is  given  from  the 
law.  A  primary  intent  of  the  law  is  to  obtain  information 
on  any  design  weaknesses  in  time  to  allow  them  to  be 
corrected.  Thus,  the  testing  required  by  the  law  is  in  some 
sense  an  aid  in  the  design  (a  discovered  weakness  can  be 
corrected)  as  well  as  a  validation  of  the  design  (if  no 
weaknesses  are  discovered,  the  design  is  presumably  vali¬ 
dated).  Analytical  vulnerability  assessments  can  assist  in 
determining  the  issues  that  require  examination  in  the  Live 
Fire  Test  program.  The  Live  Fire  Test  plan  is  developed 
using  the  information  provided  in  the  Live  Fire  Test  and 
Evaluation  Planning  Guide.  A  typical  Live  Fire  Test  plan 
will  include  early  testing  of  components,  sub-systems,  and 
sub-assemblies,  both  inert  and  full-up,  and  later  testing  of 
full-scale,  full-up  targets. 

Support  Acquisition  Decisions,  One  of  the  principal  appli¬ 
cations  of  both  analysis/modeling  and  Live  Fire  Testing  is 
to  provide  information  in  support  of  acquisition  decisions. 
This  is  accomplished  by  providing  timely  information  on 
the  vulnerability  of  the  complete  system  to  decision-making 
bodies,  such  as  the  Defense  Acquisition  Board. 

Table  1-4  presents  a  summary  of  the  applications  of  the 
analysis/modeling  methodology  and  the  Live  Fire  Testing 
methodology,  including  both  sub-scale  and  full-scale  testing. 

Previous  Studies  of  Vulnerability  Assessment  with 
Emphasis  on  Live  Fire  Testing 

Two  previous  studies  of  the  vulnerability  assessment  and 
live  fire  testing  of  military  vehicles  have  been  conducted: 
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TABLE  1-4  Applications  of  the  Methodologies 
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the  1987  U.S.  General  Accounting  Office  study  Live  Fire 
Testing,  Evaluating  DoD's  Programs  (GAO,  1987)  and  the 
1989  Board  on  Army  Science  and  Technology  (BAST), 
National  Research  Council  (NRC,  1989),  study  Armored 
Combat  Vehicle  Vulnerability  to  Anti-Armor  Weapons:  A 
Review  of  the  Army  s  Assessment  Methodology.  Both  studies 
addressed  vulnerability  issues  similar  to  those  reviewed  here. 
However,  the  GAO  study,  which  was  conducted  at  the  same 
time  the  LFT  legislation  was  enacted,  concentrated  primarily 
on  the  JLF  program.  The  purpose  of  this  study  was  to  answer 
four  questions:  (1)  What  is  the  status  of  each  system  origi¬ 
nally  scheduled  for  live-fire  testing  under  the  JLF  program? 
(2)  What  has  been  the  methodological  quality  of  the  test 
and  evaluation  process?  (3)  What  are  the  advantages  and 
limitations  of  full-up  live  fire  testing,  and  how  do  other 
methods  complement  full-up  testing?  (4)  How  can  live-fire 
testing  be  improved?  Of  interest  here  are  questions  2, 3,  and 
4. 

The  BAST  study  examined  the  Army’s  assessment  meth¬ 
odology,  including  both  analysis  and  live  fire  testing,  for 
armored  vehicles.  The  committee  conducted  an  independent 
review  to  (1)  address  issues  that  will  help  the  Army  define 
the  objectives  of  its  vulnerability  assessment  program,  (2) 
define  and  analyze  alternative  ways  to  balance  computation 
and  live  fire  testing  in  reaching  conclusions  about  vehicle 
vulnerability,  (3)  identify  technical  deficiencies  where  they 
exist,  and  (4)  suggest  alternatives  for  improvement  as  appro¬ 
priate.  All  four  tasks  are  of  interest  here. 

Although  neither  study  specifically  addressed  the  Live 
Fire  Test  legislation  and  the  DoD  LFT&E  program,  and  the 
BAST  study  did  not  consider  aircraft,  both  studies  examined 


issues  and  arrived  at  conclusions  that  are  pertinent  here. 
Furthermore,  the  personnel  and  organizations  involved  in 
the  JLF  aircraft  program  also  are  the  ones  involved  in  the 
LFT  aircraft  program.  Consequently,  the  major  issues  and 
conclusions  of  these  two  studies  as  they  apply  to  aircraft 
are  presented  in  Appendixes  B  and  C. 
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Evaluation  of  the  Cost, 
Effectiveness,  and  Deficiencies  of 
These  Methodologies 


Regardless  of  the  specific  application(s)  of  a  vulnerability 
assessment  of  an  aircraft,  whether  it  is  to  aid  in  design 
or  design  validation,  satisfy  program  requirements,  support 
subsequent  analytical  assessments,  predict  test  outcomes, 
satisfy  the  Live  Fire  Test  law,  or  support  acquisition  deci¬ 
sions,  the  goal  of  the  assessment  is  to  obtain  information 
on  the  vulnerability  of  that  aircraft.  This  information  can  be 
described  in  terms  of  the  attributes  of  the  types,  amounts, 
and  applications  of  the  information  obtained;  the  accuracy 
of,  or  level  of  confidence  in,  the  information;  and  the  cost 
required  to  obtain  the  information.  An  effective  methodology 
is  one  that  provides  a  great  deal  of  accurate  information,  of 
all  types,  with  many  applications,  at  very  low  cost,  and  with 
very  few  deficiencies.  Both  of  the  methodologies  reviewed 
in  Chapter  1  (analysis/modeling  and  live  fire  testing1)  indi¬ 
vidually  have  certain  advantages  and  disadvantages  with 
respect  to  accomplishing  the  goal  of  obtaining  information 
on  the  vulnerability  of  aircraft 

In  general,  the  analysis/modeling  methodology  can  pro¬ 
vide  considerable  numerical  information  on  the  vulnerabil¬ 
ity  of  the  total  aircraft  from  all  aspects  for  all  threats  at  a 
reasonable  cost  but  the  level  of  confidence  placed  in  the 
information  can  vary  from  very  low  to  high,  depending 
upon  the  type  of  information  obtained,  the  model  used,  the 


'The  reader  is  reminded  of  the  difference  between  general  live  fire 
testing  and  the  congressionally  mandated  Live  Fire  Testing. 


quality  of  input  data,  the  analyst  and  the  evaluator.2  On 
the  other  hand,  although  the  live  fire  testing  methodology 
has  the  potential  to  obtain  data  that  can  be  used  to  determine 
numerical  information  on  the  vulnerability  of  the  aircraft 
to  a  particular  weapon  for  many  hits  over  the  entire  pre¬ 
sented  area  of  the  aircraft  from  all  aspects,  in  actual  practice, 
testing  provides  information  only  on  the  aircraft’s  vulnera¬ 
bility  to  hits  in  relatively  few  locations,3  and  the  expenditure 
of  funds  required  to  obtain  this  information  is  relatively 
large.  However,  the  level  of  confidence  in  the  test  results 
usually  is  relatively  high.  This  chapter  examines  the  three 
information  attributes  of  types  (including  amounts  and  ap¬ 
plications),  accuracy,  and  cost  for  both  methodologies. 

Analysis/Modeling 

Type,  Amount,  and  Applications  of  the  Information .  The 
analysis/modeling  programs  described  in  Chapter  1  for  the 


The  committee  notes  that  there  are  strong  differences  in  opinions 
concerning  the  level  of  confidence  to  be  placed  in  the  analytical  results. 
Some  consider  them  to  be  very  accurate,  whereas  others  believe  that  there 
are  few,  if  any,  analytical  results  that  are  accurate.  However,  there  is 
consensus  in  the  committee  that  the  present  analytical  models  are  flawed 
in  several  ways  and  are  weak  in  their  ability  to  extrapolate  beyond  thetr 
existing  data  base. 

These  few  locations  are  selected  to  maximize  the  amount  of  useful 
data  obtained  and  to  answer  specific  questions  concerning  the  aircraft's 
vulnerability. 
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three  types  of  weapons  provide  numerical  values  for  the 
vulnerability  of  the  individual  critical  components  and  the 
total  aircraft  from  all  aspects  around  the  aircraft  and  for  any 
selected  ballistic  projectile  or  guided  missile.  The  numerical 
values  can  be  in  the  form  of  vulnerable  areas  or  probabilities 
of  kill.  This  information  can  be  used  to  aid  in  design  and 
design  validation,  to  satisfy  program  requirements,  to  sup¬ 
port  subsequent  analytical  assessments,  to  predict  test  out¬ 
comes,  and  to  support  acquisition  decisions. 

Accuracy  of  Analytical  Models .  There  are  three  aspects 
of  the  accuracy  of  the  analytical  models  that  need  to  be 
examined:  model  verification  (Are  the  internal  workings  of 
the  model  in  good  order — are  the  equations  properly 
coded?),  model  validation  (Does  the  model  adequately  repre¬ 
sent  the  processes  it  portrays— do  the  equations  adequately 
represent  the  actual  physical  situation?),  and  model  accredi¬ 
tation  (Is  the  model  appropriate  to  the  particular  applica¬ 
tion — is  the  model  capable  of  properly  representing  the  air¬ 
craft  and  the  weapon?). 

Model  Verification .  In  so  far  as  model  verification  is  con¬ 
cerned,  there  appears  to  be  little  reason  to  doubt  the  veracity 
of  the  logic  and  coding  of  the  models  described  in  Chapter 
1.  They  have  been  widely  employed  by  government  labora¬ 
tories  in  all  three  military  Services  and  by  a  great  many 
nongovernmental  users.  The  committee  accepts  the  fact  that 
the  internal  workings  of  the  models  are  in  good  order. 

Model  Validation.  Credibility  problems  exist  with  model 
validation.  Despite  all  the  physical  phenomena  that  the  cur¬ 
rent  aircraft  vulnerability  models  attempt  to  depict  (and 
many  are  depicted  with  high  confidence),  there  are  some 
phenomena  that  are  known  to  exist,  but  have  not  been  charac¬ 
terized  and  entered  into  the  model  structure,  or  are  poorly 
modeled.  Perhaps  the  most  notable  phenomenon  not  mod¬ 
eled  is  the  random  deviation  or  ricochet  of  the  path  of  the 
penetrator  or  fragment  from  the  assumed  straight  shotline 
as  it  passes  through  the  aircraft  components.  Another  phe¬ 
nomenon  not  modeled  is  the  synergism  that  occurs  when  a 
portion  of  an  aircraft  is  hit  by  a  multitude  of  closely  spaced 
fragments.  In  this  multiple,  closely  spaced  hit  condition, 
the  fragments  can  be  far  more  damaging  to  the  impacted 
structure  than  when  they  are  not  closely  spaced.  A  third 
example  of  a  phenomenon  not  explicitly  modeled  is  spall. 
The  backface  spall  generated  by  the  impact  of  a  projectile 
or  fragment  on  a  plate  is  not  considered  as  additional  frag¬ 
ments  to  be  tracked  through  the  aircraft  by  the  shotline 
model.4 


t  *. 

4SpaU  that  is  generated  within  a  component  is  implicitly  accounted  for 
in  terms  of  its  damage  to  the;  component  since  the  empirically  determined 
total  damage  is  the  sum  of  all  of  the  damage  mechanisms  and  processes 
within  the  component,  including  the  internal  spall. 


A  poorly  modeled  phenomenon  is  the  treatment  of  the 
remaining  parts  of  an  impacting  fragment  or  penetrator  after 
it  penetrates  a  plate.  The  current  model  considers  only  the 
largest  remaining  piece  in  calculating  subsequent  effects. 
Another  poorly  modeled  phenomenon  is  the  increase  in  a 
component’s  P^,  that  occurs  when  the  component  is  hit  by 
more  than  one  projectile  or  fragment  or  is  damaged  by  both 
blast  and  fragments.  The  assumption  is  nearly  always  made 
that  the  same  P^,  value  for  the  first  hit  in  the  component  is 
adequate  for  subsequent  impacts  on  the  component.  The 
increase  in  a  component’s  vulnerability  due  to  damage 
caused  by  prior  hits  is  neglected.  For  example,  a  tail  rotor 
drive  shaft  on  a  helicopter  may  not  be  killed  when  first  hit 
by  a  tumbled  12.7-millimeter  armor-piercing  projectile  with 
incendiaries  (API),  but  a  second  hit  in  the  vicinity  of  the 
first  hit  may  cause  the  shaft  to  break  due  to  the  synergism 
in  damage  between  projectiles.  This  increase  in  P^,  due  to 
previous  hits  is  usually  neglected. 

Also,  there  are  known  physical  phenomena  that  are  mod¬ 
eled,  but  their  numerical  values  are  not  well  known  because 
they  have  not  been  tested,  or  the  test  results  are  for  conditions 
not  satisfied  in  the  combat  incident.  For  example,  the  pene¬ 
tration  equations  that  determine  the  velocity  and  mass  decay 
of  fragments  and  penetrators  as  they  penetrate  plates  on  the 
aircraft  may  not  have  the  proper  decay  coefficients  for  the 
material  of  interest.  Furthermore,  these  equations  have  been 
developed  for  specific  geometric  shapes  of  impactors,  such 
as  spheres  and  cubes,  whereas  fragments  are  usually  irregu¬ 
lar.  Two  of  the  most  difficult  kill  modes  to  model  in  aircraft 
are  fire  and  explosion.  This  is  due  to  the  randomness  of  fuel 
sloshing  within  a  tank,  fuel  leakage  into  dry  bays  around 
the  tanks,  and  fuel  migration  into  distant  portions  of  the 
aircraft. 


S  e  gleet  of  On-Board  Ordnance.  One  of  the  most  important 
findings  of  our  study  is  that  on-board  ordnance  is  usually 
neglected  as  a  contributor  to  vulnerability.  Most  of  the  simu¬ 
lation  of  internal  ordnance  aboard  the  aircraft  has  been 
treated  by  the  aircraft  vulnerability  community  as  “clutter.” 
Clutter  is  inert  material  in  a  compartment  that  is  considered 
only  as  a  compartment  filler  in  the  calculation  of  overpres¬ 
sure  in  the  compartment  due  to  its  volume.  The  overpressure 
from  the  explosive  “clutter”  when  hit  by  projectiles  and 
fragments  can  change  the  total  pressure  in  the  compartment 
dramatically.  One  of  the  basic  premises  of  all  development 
testing  is  that  modeling  must  precede  testing.  Any  aspect 
of  hardware  impact  on  vulnerability  must  first  be  modeled 
so  that  the  testing  can  be  used  to  verify  the  modeling  rather 
than  for  the  testing  to  be  extensive  enough  to  cover  all 
statistical  events.  One  of  the  basic  requirements  of  the  Live 
Fire  Test  program  is  to  test  full-scale  vehicles  with  a  full 
load  of  on-board  ordnance.  The  continual  neglect  of  one  of 
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the  basic  vulnerability  contributors  will  make  it  more  diffi¬ 
cult  to  convince  anyone  that  computer  modeling  can  be 
substituted  for  the  full-up  testing  of  combat-loaded  aircraft. 

Weapon  Lethality  Assessments  V ersus  Aircraft  Vulnerabil¬ 
ity  Assessments .  Some  of  the  programs  currently  used  to 
compute  an  aircraft's  vulnerability  to  the  various  types  of 
weapons  were  developed  by  the  Joint  Technical  Coordinat¬ 
ing  Group  for  Munitions  Effectiveness.  Because  the  muni¬ 
tions  effectiveness  community  wanted  conservative  esti¬ 
mates  for  the  prediction  of  a  weapon’s  lethality,  its  programs 
were  developed  so  that  our  weapons  were  not  to  be  credited 
with  any  kill  capability  that  was  not  clearly  justified;  a  sound 
policy  from  the  weapon  development  point  of  view.  How¬ 
ever,  as  a  result  of  this  approach,  the  use  of  these  programs 
to  predict  the  vulnerability  of  a  U.S.  aircraft  most  likely 
leads  to  overly  optimistic  predictions.  Potential  vulnerabili¬ 
ties  have  been  ignored  unless  clearly  justified;  an  unsound 
policy  where  the  survivability  of  U.S.  aircrews  is  concerned. 
Examples  of  where  the  vulnerability  of  aircraft  is  underesti¬ 
mated  are 

•  the  use  of  the  Thor  penetration  equations,  which  con¬ 
sider  only  the  largest  remaining  piece  of  a  penetrator  as  it 
passes  through  the  aircraft, 

•  the  lack  of  synergism  due  to  both  multiple  simultane¬ 
ous  and  sequential  hits  on  a  component; 

•  the  lack  of  direct  consideration  of  spall,  and 

•  the  lack  of  consideration  of  cascading  damage,  such 
as  fuel  tank  damage  leading  to  fuel  ingestion. 

The  Joint  Technical  Coordinating  Group  on  Aircraft  Surviv¬ 
ability  (JTCGIAS)  Pm  Workshop .  Even  if  all  other  aspects 
of  the  model  were  perfect,  some  of  those  portions  of  the 
model  that  portray  the  results  of  the  physical  interactions  or 
damage  processes  that  occur  when  a  threat  weapon  interacts 
with  an  aircraft  are  inaccurate  or  incomplete.  In  order  to 
eliminate  some  of  the  model  deficiencies,  more  vulnerability 
data  are  needed  on  component  kill  modes  and  functions. 
This  deficiency  in  the  component  vulnerability  data  base 
was  the  subject  of  a  JTCG/AS  Component  Pw  Workshop 
held  from  March  5-8,  1991,  at  Wright-Patterson  Air  Force 
Base  (WPAFB). 

The  objectives  of  the  JTCG/AS  Workshop  were  to  criti¬ 
cally  review  the  current  state-of-the-art  of  component  Pw 
prediction,  to  recommend  a  set  of  P*  or  values  or  func¬ 
tions  for  use  in  analyses,  and  to  develop  plans  for  improving 
and  validating  this  set.5  Working  panels  were  organized  as 
follows: 


’Component  vulnerability  is  sometimes  quantified  by  P**,  the  probability 
the  component  is  damaged  given  a  hit 


Fuel  System 

Flight  Controls/Hydraulics  (Control  and  Power) 

Crew  Station 

Engines  and  Accessories 

Stores,  Ammunition,  and  Flares 

Electrical  and  Avionics 

Structures, 'Landing  Gear,  and  Armor 

Helicopter  Unique  Components 

The  draft  repons  of  the  individual  panels  are  currently 
being  integrated  into  a  final  repon.  The  findings  of  the  eight 
working  panels  are  given  in  Appendix  D.6 

The  general  conclusion  from  the  workshop  is  that  the 
component  vulnerability  data  base  is  the  weakest  link  in  the 
vulnerability  analysis  chain.  Clearly,  much  remains  to  be 
done  with  respect  to  validating  those  sections  of  the  aircraft 
vulnerability  models  that  deal  with  component  and  subsys¬ 
tem  damage  prediction.  The  level  of  confidence  placed  in 
the  analytical  models  would  be  significantly  increased  if  a 
concerted  effort  was  made  to  determine  the  maximum  error 
in  Av  that  occurs  in  the  model  predictions  using  the  current 
data  base,  and  to  what  extent  this  error  might  be  reduced 
with  the  availability  of  new  test  data. 

Model  Accreditation,  Model  accreditation  is  accomplished 
by  the  JTCG/AS.  This  organization,  consisting  of  representa¬ 
tives  from  all  three  Services,  has  established  procedures  for 
verifying,  validating,  and  accrediting  models.  Despite  the 
deficiencies  identified  by  the  aircraft  vulnerability  assess¬ 
ment  community  in  its  analytical  models,  the  community 
has  been  exemplary  in  the  exchange  of  data  and  ideas  among 
the  three  Services  and  industry,  in  the  development  of  hand¬ 
books  and  design  guides  for  reducing  vulnerability,  and  in 
the  establishment  of  validated  data  bases  from  test  data. 
Although  there  is  still  much  to  learn,  more  perhaps  is  known. 
There  will  always  be  a  need  to  update  models  with  new 
input  data  as  aircraft  materials  and  designs  change  and  as 
the  threat  weapons  change. 

The  Cost  of  Analysis.  It  is  extremely  difficult  to  determine 
the  costs  associated  with  a  typical  analytical  assessment 
since  the  cost  depends  heavily  on  the  type  of  aircraft  being 
assessed  and  the  particular  application  for  the  assessment. 
However,  some  rough  estimates  for  the  costs  to  conduct  an 
analysis  that  would  be  appropriate  for  a  major  milestone 
decision  are  given  below.  The  numbers  were  obtained  in 
a  personal  communication  from  a  representative  from  the 
Ballistic  Research  Laboratory. 


These  findings  are  taken  from  a  preliminary  copy  of  a  briefing  being 
prepared  to  be  given  to  the  JTCG/AS  Central  Office  and  to  the  Office  a i 
the  Secretary  of  Defense.  This  briefing  material  was  provided  by  Gerald 
Bennett,  ASD/XRM,  WPAFB. 
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Cost  of  the  Model.  The  cost  of  creating  a  CO V  ART  or 
HEVART  model  for  a  helicopter  is  roughly  between  575,000 
and  5200,000,  with  the  low  end  for  an  upgrade  to  an  existing 
system  and  the  high  end  for  an  all  new,  high  detail  description 
of  a  U.S.  system.  These  costs  also  include  the  generation 
of  the  shotline  data  files  needed  to  run  the  Computation  of 
Vulnerable  Area  and  Repair  Time  Model  (CO V ART)  or 
High  Explosive  Vulnerable  Area  and  Repair  Time  (HEV¬ 
ART)  model.  The  cost  of  developing  appropriate  target 
models  for  the  Stochastic  Qualitative  Analysis  of  System 
Hierarchies  (SQuASH)  might  be  two  to  three  times  that 
required  for  the  conventional  vulnerable  area  models.7  As 
the  design  community  increases  its  use  of  three-dimensional 
modeling  and  grid  generation,  and  as  computer  usage  be¬ 
comes  less  expensive,  the  cost  of  modeling  will  come  down 
significantly.  The  major  cost  will  be  the  preparation  of  input 
data. 

Cost  of  Computer  Runs.  A  typical  batch  of  computer  runs 
for  a  helicopter  includes  two  flight  modes  (hover  and  forward 
flight)  and  from  6  to  26  attack  aspects.  The  cost  of  this  batch 
of  runs  using  COVART  or  HEVART  ranges  from  545,000 
to  5100,000.  The  analysis  and  preparation  of  input  data 
account  for  most  of  this  cost. 

Cost  of  Obtaining  Supporting  Experimental  Data  for  PiA 
Functions .  The  execution  of  an  analytical  model  without 
any  live  fire  test  data  on  kill  modes  and  P^,  functions  as  input 
data  on  component  vulnerability  could  be  accomplished,  but 
the  level  of  confidence  in  the  results  would  be  very  low. 
Consequently,  the  cost  of  obtaining  the  necessary  data  on 
component  vulnerability  must  be  included  in  the  cost  of  the 
analysis.  If  all  of  these  data  are  available  from  prior  tests, 
the  cost  of  gathering  them  is  relatively  low.  If  all  of  the 
necessary  data  are  not  available,  supporting  tests  must  be 
carried  out  to  obtain  the  missing  data.  These  tests  can  be  as 
simple  as  the  firing  of  fragments  at  pieces  of  plate  to  measure 
penetration  or  V*  velocities,  or  they  can  be  as  complicated 
as  firing  several  fragments  at  running  engines  to  determine 
the  severity  of  damage,  or  firing  at  major  portions  of  the 
aircraft  structure  that  contains  fuel  tanks  to  simulate  the  fire/ 
explosion  and  hydraulic  ram  phenomena.  The  typical  costs 
associated  with  two  types  of  live  fire  tests  are  given  below. 

For  a  helicopter  engine  and  a  23-millimeter  HEI  threat, 
basic  preliminary  testing  with  components  and/or  a  static 
engine  would  cost  approximately  530,000.  Comprehensive 
testing  with  a  full-up  running  engine  would  cost  approxi¬ 
mately  $250,000.  For  a  tail  boom  structure,  basic  shoot-and- 
look  damage  characterization  tests  would  cost  approximately 
525,000.  A  comprehensive  evaluation  of  the  test  results. 


7Note  that  SQuASH  has  not  yet  been  applied  to  aircraft  vulnerability 
assessment. 
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including  postdamage  controlled  structural  experiments 
would  cost  approximately  5150,000. 

Live  Fire  Testing 

A  live  fire  test  is  one  in  which  live  ammunition,  either 
explosive  or  non-explosive,  is  fired  at  a  target.  Live  fire 
testing  can  be  used  to  assist  in  the  design  and  design  valida¬ 
tion  of  an  aircraft,  to  provide  empirical  information  on  com¬ 
ponent  vulnerability  in  support  of  the  analytical  models,  to 
satisfy  the  Live  Fire  Test  law,  and  to  support  acquisition 
decisions.  Live  fire  testing  is  part  of  either  developmental 
testing  (DT)  or  Live  Fire  Testing.  The  difference  between 
the  two  types  of  tests  is  that  DT  is  part  of  the  normal  design 
and  development  process,  whereas  Live  Fire  Testing  is  that 
testing  intended  to  satisfy  the  Live  Fire  Test  law.  “As  cur¬ 
rently  defined,  development  test  and  evaluation  (DT&E)  is 
that  test  and  evaluation  (T&E)  conducted  throughout  the 
acquisition  process  to  assist  in  the  engineering  design  and 
development  process  and  to  verify  attainment  of  technical 
performance  specifications  and  objectives  and  supportabil- 
ity.  DT&E  includes  T&E  of  components,  computer  software, 
subsystems,  and  hardware/software  integration.  It  encom¬ 
passes  the  use  of  modeling,  simulations,  and  test  beds,  as 
well  as  advance  development,  prototype,  and  full-scale  engi¬ 
neering  development  models  of  the  system.  Technical  per¬ 
formance  specifications  must  be  validated  through  DT&E 
in  order  for  the  developer  (program  manager)  to  certify  that 
the  weapon  system  is  ready  for  the  final  phase  of  Initial 
Operational  Test  and  Evaluation  (IOT&E)”  (OSD,  1987). 

Of  particular  interest  here  is  the  specific  test  and  evalua¬ 
tion  methodology  required  to  satisfy  the  Live  Fire  Test  (LFT) 
legislation.  The  law  requires  realistic  survivability  (vulnera¬ 
bility)  testing.  Although  such  a  test  program  will  in  fact 
involve  early  component  and  subsystem  or  sub-scale  Live 
Fire  Testing,  the  hallmark  of  this  approach  is  a  substantial 
test  program  that  involves  a  significant  number  of  shots 
against  a  combat-configured,  full-scale  version  of  the 
weapon  system  (i.e.,  a  full-scale  LFT).  The  early  Live  Fire 
Tests  on  components  and  subsystems  provide  information 
on  any  vulnerabilities  of  the  individual  components  and 
subsystems.  Once  the  information  from  these  tests  has  been 
evaluated,  the  tests  on  the  full-scale  aircraft  are  to  be  con¬ 
ducted  as  mandated  by  the  law,  unless  a  waiver  has  been 
granted.  The  full-scale  test  program  will  often  include  a 
number  of  shots  that  are  randomly  chosen  and  a  number  of 
shots  that  are  selected  to  address  specific  issues.  Real  (or 
realistic  surrogates  of)  threats  likely  to  be  encountered  in 
combat  must  be  used  in  the  tests.  These  threats  can  be 
non-explosive  ballistic  projectiles,  ballistic  projectiles  with 
contact-fuzed  and  proximity-fuzed  high-explosive  (HE)  war¬ 
heads,  and  guided  missiles  with  contact-fuzed  and  proximi¬ 
ty-fuzed  HE  warheads.  The  three  information  attributes  of 
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types  (including  amounts  and  applications),  accuracy,  and 
cost  for  both  sub-scale  testing  and  full-scale  testing  are  exam¬ 
ined  below. 


Type,  Amount ,  and  Applications  of  the  Information .  In 
general,  the  Live  Fire  Tests  on  both  sub-scale  and  full-scale 
targets  produce  information  on  what  actually  happened  for 
a  particular  set  of  test  conditions  (e.g.,  the  specific  target, 
weapon,  and  shotline)  and  typically  for  a  relatively  small 
number  of  shots  under  these  conditions.8  Some  typical  exam¬ 
ples  of  tests  on  sub-scale  targets  are  tests  to  determine  the 
penetration  capability  of  fragments  through  plates  of  com¬ 
posite  materials,  tests  on  helicopter  rotor  blades,  tail  booms, 
and  gear  boxes  using  small  arms  projectiles  and  small-caliber 
HE  AAA  rounds,  tests  on  fuel  tank  simulators  using  small 
caliber  HE  AAA  to  determine  the  efficacy  of  a  particular 
fuel  tank  protection  scheme,  and  stand-alone  tests  on  running 
engines  using  several  impacting  fragments.  The  information 
from  these  tests  ranges  from  measured  fragment  velocities, 
temperatures,  and  overpressures  to  the  ability  of  the  tested 
article  to  continue  to  function  after  the  hit.  Tests  on  the  full- 
scale  aircraft  will  most  likely  be  conducted  with  the  aircraft 
on  the  ground  or  suspended;  it  can  neither  crash  nor  be 
forced  to  land  as  a  result  of  the  shot.  Thus,  the  ability  of 
the  aircraft  to  sustain  the  essential  functions  for  flight  after 
the  hit  is  not  observed  directly.  Furthermore,  rather  than 
produce  a  sufficient  amount  of  numerical  data  from  the  full- 
scale  test  that  can  be  used  directly  to  determine  the  kill 
probability  of  the  aircraft  for  the  shot,  each  full-scale  test 
provides  a  list  of  damaged  components  along  with  descrip¬ 
tions  of  the  details  of  the  extent  and  severity  of  the  damage 
and  the  associated  damage  events  for  each  shot. 

With  respect  to  the  applications  of  the  information  ob¬ 
tained  from  the  tests,  the  committee  notes  that  the  Live  Fire 
Tests  are  conducted  primarily  to  ( 1)  satisfy  the  LFT  law  and 
its  intent  (i.e.,  to  determine  any  inherent  vulnerabilities  in 
the  design  sufficiently  early  in  the  program  to  allow  the 
vulnerabilities  to  be  corrected),  and  (2)  provide  information 
in  support  of  acquisition  decisions.  They  are  not  specifically 
intended  to  provide  information  that  can  be  used  to  improve 
the  analytical  models  for  predicting  aircraft  vulnerability. 
Nevertheless,  previous  experience  with  full-scale  Live  Fire 
Tests  on  ground  vehicles  has  shown  that  important  types  of 
damage  and  kill  modes  have  been  observed  that  were  not 
included  in  the  analytical  models  for  these  vehicles.  Thus, 
the  information  provided  by  these  LFTs  can  be  used  in  the 
other  applications.  Besides  satisfying  the  letter  of  the  law, 
the  LFT&E  program  provides  (in  principle)  an  opportunity 
to  !  « 


•Typically,  the  smaller  the  target  the  more  greater  is  the  number  of 
test  shots,  and  the  more  detailed  is  the  information. 


•  obtain  information  on  the  vulnerability  of  the  aircraft; 

•  find  vulnerabilities  that  were  not  anticipated  by  the 
analyses;  and 

•  gather  data  on  the  synergism  among  the  different  dam¬ 
age  processes  and  kill  modes. 

All  three  aid  in  the  design  and  design  validation,  support 
the  analytical  models,  and  support  acquisition  decisions. 

Accuracy  of  and  Level  of  Confidence  in  the  Informa • 
tion .  The  level  of  confidence  that  is  placed  in  the  sub¬ 
scale  and  full-scale  Live  Fire  Test  results  depends  primarily 
upon  the  level  of  realism  of  the  tests.  Certainly,  there  is  a 
possibility  for  realism  in  Live  Fire  Testing.  Some  of  the 
vulnerability  aspects  that  are  represented  in  Live  Fire  Testing 
are  the  effects  of  blast,  including  structural  deformation  and 
component  damage,  and  component  damage  due  to  multiple 
fragment  hits,  including  bending,  breaking,  and  perforating. 
Other  aspects  of  vulnerability  that  are  represented  include  the 
occurrence  of  spall  and  the  penetration  through  components. 
The  penetration  damage  and  velocity  decay  in  the  test  are 
the  result  of  real  penetrators  going  through  real  materials, 
with  no  assumptions  about  breakup,  ricochet,  etc.9  Syner- 
gisms  among  blast,  fire,  spall,  and  fragments  are  realistically 
represented  to  the  extent  that  the  other  aspects  of  the  test 
conditions  (e.g.,  air  speed  and  altitude)  are  adequately  repre¬ 
sented. 

Although  full-scale  tests  Have  the  ^potential  to  provide  the 
most  realism,  there  are  some  problemsTt>articularly  with  the 
weapons  used,  the  flight  conditions,  crew  vulnerability,  and 
on-board  munitions.  The  weapons  selected  for  testing  will 
most  likely  be  those  that  are  not  overmatching  (i.e.,  they 
will  not  have  a  high  probability  of  destroying  the  aircraft). 
Consequently,  most  of  the  weapons  will  be  either  non-explo¬ 
sive  or  small-caliber  explosive  rounds.  When  larger  explo¬ 
sive  weapons  are  used,  particularly  the  large-caliber  projec¬ 
tiles  and  guided  missiles,  the  damage  to  the  aircraft  can  be 
severe  and  widespread,  making  it  very  difficult  to  repair  the 
aircraft  and  return  it  to  a  condition  that  would  be  satisfactory 
for  further  testing.  Associated  with  the  explosive  weapon  is 
the  location  of  the  point  of  detonation.  Contact-fuzed  HE 
warheads  must  impact  the  aircraft  in  order  to  damage  it,  but 
proximity-fuzed  weapons  can  detonate  at  distances  ranging 
from  the  aircraft  skin  to  several  hundred  feet  away.  If  the 
warhead  is  detonated  too  close  to  the  aircraft,  it  could  destroy 
it. 

With  respect  to  the  flight  conditions,  airflow  can  be  simu¬ 
lated  to  some  extent,  although  it  seems  unlikely  that  an  entire 


’Although  penetration  is  realistically  represented  in  the  test,  the  informa¬ 
tion  obtained  from  the  test  most  likely  will  not  provide  penetration  data 
per  se  for  the  analytical  models  because  there  will  be  few  if  any  data  on 
the  parameters  associated  with  the  individual  penetrations,  such  as  impact 
and  exit  velocities,  masses,  and  angles. 
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transport  aircraft  would  be  placed  in  a  uniform  high-speed 
airstream.  Static  loading  conditions  can  be  simulated,  at  least 
over  a  part  of  the  aircraft,  but  there  is  some  concern  about 
the  effects  of  wind  gusts  and  about  transient  loads  produced 
by  maneuvering  and  loss  of  control.  Furthermore,  altitude 
and  temperature  are  not  simulated.  For  externally  detonating 
warheads,  a  static  test  may  result  in  a  different  set  of  blast 
and  fragment  impact  conditions  from  those  of  a  dynamic 
test.  In  some  warhead/target  encounters  (e.g.,  in  head-on 
pass  or  when  overtaking  at  very  high  velocity),  the  high- 
velocity  fragments  may  impact  on  one  part  of  the  target 
aircraft’s  structure  while  the  slower  blast  may  affect  a  differ¬ 
ent  area.  In  static  tests,  both  affect  the  same  portion  of  the 
target. 

Clearly,  real  people  cannot  be  used  in  the  tests.  Anthropo¬ 
morphic  dummies,  pressure  gauges,  and  gas-sampling  equip¬ 
ment  can  be  used  to  obtain  data  on  vulnerability  issues 
related  to  personnel  vulnerability.  On  the  other  hand,  the 
responses  of  the  crew  to  shock,  incapacitation,  temporary 
loss  of  control,  etc.,  are  not  directly  measured  and  must  be 
inferred.  These  responses  are  critical  to  both  crew  and  air- 
craft  survivability.10 

Vulnerability  of  the  Aircraft.  Although  it  is  true  that  certain 
catastrophic  kills,  such  as  an  explosion  within  an  aircraft 
fuel  tank,  would  be  observable  in  a  test,  it  is  also  true  that 
other  types  of  kills  would  not  be  directly  observable.  For 
example,  would  the  aircraft  actually  crash  after  damage  to 
one  of  the  control  surfaces?  Even  if  the  actual  kill  was 
directly  observed,  there  are  the  problems  of  associating  the 
results  with  the  other  kill  categories  and  levels,  and  of  extrap¬ 
olating  the  results  to  other  threats  and  tactical  conditions. 
For  example,  suppose  the  proximity-fuzed  detonation  of  an 
85-millimeter  HE  warhead  near  the  left  differential  stabilator 
of  the  aircraft  removed  90%  of  the  stabilator.  Is  this  a  kill,  and 
if  it  is  for  the  85-millimeter  weapon,  would  a  57-millimeter 
warhead  detonation  in  the  same  location  cause  the  same 
kill? 


Unanticipated  Vulnerabilities.  TOe  analytic  models  are 
presently  structured  to  provide  aircraft  kill  pmbabJmes 
based  upon  assumed  kill  modes,  and  an  aircraft  with  reduced 
vulnerability  is  designed  to  prevent  these  kill  modes  from 
occurring.  However,  if  a  kill  mode  is  unanticipated  m  both 
the  analysis  and  the  design,  the  model  will  underestimate 
the  aircraft’s  actual  vulnerability,  and  the  aircraft  will  contain 
this  vulnerability.  These  unanticipated  kill  modes  may  be 
local  effects  that  occur  in  the  vicinity  of  the  original  impact, 
or  they  may  be  caused  by  cascading  damage  from  the  impact 
location  to  a  distant  part  of  the  aircraft  For  a  local  effect 


'*me  committee  now  that  these  deputies  from  realism  in  the  live 
fire  tests  are  more  severe  in  analysis/tnodeling. 


example,  suppose  the  unanticipated  kill  mode  was  an  electri¬ 
cal  wire  bundle  that  caught  fire  when  hit  by  a  bullet  or 
fragment.  This  is  a  local  kill  mode  that  could  be  discovered 
by  component  or  subsystem  testing.  On  the  other  hand, 
suppose  the  unanticipated  kill  mode  was  a  fuel  ingestion 
kill  of  an  engine  mounted  externally  on  the  rear  portion  of 
the  fuselage.  Hydraulic  ram  pressure  in  the  fuel  in  the  wing 
fuel  tank  in  front  of  the  engine  (due  to  a  penetrator)  caused 
fuel  to  spew  out  the  top  of  the  tank.  This  fuel  was  ingested 
by  the  engine,  which  then  died.  A  test  of  only  the  wing  fuel 
tank  might  not  reveal  this  kill  mode  because  of  the  absence 
of  the  running  engine. 

Although  a  full-scale  Live  Fire  Test  might  reveal  a  local 
unanticipated  kill  mode,  such  as  the  burning  wire  bundle, 
full-scale  testing  is  not  necessarily  an  efficient  methodology 
for  obtaining  this  information.  Furthermore,  one  can  not  say 
with  great  confidence  that  if  no  unexpected  vulnerabilities 
occurred  in  the  full-scale  Live  Fire  Tests,  then  there  are 
none  to  be  discovered  later  in  combat  Kill  modes  involving 
cascading  damage  may  not  always  occur  in  a  test,  and  they 
may  be  particularly  difficult  to  observe  in  a  full-scale  test 
if  they  do  occur.1'  Some  kill  modes  due  to  cascading  effects 
are  well  known,  such  as  the  kill  of  an  engine  due  to  the 
ingestion  of  fuel  from  a  damaged  fuel  tank  next  to  an  air 
inlet,  and  are  relatively  easy  to  observe.  Others,  such  as  the 
migration  of  toxic  fumes  or  flames  from  one  portion  of  the 
aircraft  to  another,  are  not  as  well  known,  may  not  always 
occur,  and  may  be  difficult  to  detect  if  they  do  occur. 

It  is  not  feasible  to  test  all  possible  combat  situations  for 
unanticipated  vulnerabilities  using  full-scale  aircraft  because 
of  the  large  number  of  parameters  that  affect  the  target  s 
vulnerability.  These  parameters  include  all  of  the  weapons 
likely  to  be  encountered  in  combat,  the  tactical  situations 
of  interest,  and  all  of  the  possible  impact  locations  on  the 
aircraft  (i.e.,  the  shotlines).  Furthermore,  the  damaged  air¬ 
craft  should  be  returned  to  its  original  condition  after  each 
test  if  appropriate  and  possible.  Consequently,  the  test  plan 
may  contain  a  number  of  random  shots,  a  number  of  random 
shots  from  directions  expected  in  combat,  a  number  of  se- 
lected  shots,  and  an  ordering  that  schedules  potentially  cata¬ 
strophic  situations  at  the  end  of  the  program.11 

Likelihood  of  Discovering  a  Particular  Vulnerability.  Un¬ 
less  a  particular  vulnerability  is  relatively  insensitive  to  the 
parameters  associated  with  a  large  subset  of  the  conditions 
and  has  a  relatively  high  probability  of  occurring,  it  may 
remain  undiscovered  in  a  test  program.  For  example,  sup¬ 
pose  that  a  particular  vulnerability  event  associated  with  a 


"This  is  the  reason  that  Live  Fire  Test  programs  require  some  random 
hit,  oo  incidents  dut  occur  relatively  frequently  rather 

thin  on  ail  unanticipated  kill  modes. 

'The  number  of  shotlines  in  the  vulnerability  analysts  of  the  F-22  as 

approximately  300.000  per  threat. 
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FIGURE  2-1  Relationship  among  the  probability  of  occurrence  on  each  test,  the  number  of  tests,  and  the  probability  of  at  least  one 
observation  over  the  number  of  tests. 


high-explosive  weapon  had  a  0.2  probability  of  occurrence 
on  any  random  shot  for  ail  test  conditions  of  interest.  If  the 
test  plan  consisted  of  10  random  shots  around  the  aircraft, 
the  probability  the  event  would  occur  at  least  once  in  the 
10  tests  is  1  -  (1  -  0.2)10  =  0.89  and  hence  the  probability 
it  would  not  occur  is  0.11.  Thus,  this  vulnerability  would 
most  likely  be  observed.13  On  the  other  hand,  if  the  event 
could  only  occur  on  2  of  the  10  random  shots  with  a  0.2 
probability  (and  a  probability  of  zero  on  the  other  8  shots), 
the  probability  the  event  would  occur  at  least  once  is 

1  -  (I  -  0.2)2  -  0.36 

and  the  probability  that  it  would  not  occur  is  0.64.  Thus,  this 
particular  vulnerability  would  most  likely  not  be  observed. 

Figure  2-1  shows  the  relationship  among  the  probability 
of  occurrence;  the  probability  of  observation  of  0.25,  0.50, 


uOther  known  vulnerabilities  may  also  occur  during  the  tests  that  could 
obscure  the  particular  vulnerability,  and  other  unknown  vulnerabilities  could 
also  occur. 


0.75,  or  0.95;  and  the  number  of  tests.  This  figure  can  be 
used  to  determine  the  number  of  tests  required  to  obtain  a 
probability  of  observation  of  0.25,  0.50,  0.75,  or  0.95  for  a 
given  probability  of  occurrence.  For  example,  if  the  probabil¬ 
ity  of  occurrence  in  each  test  is  0.2  and  the  desired  probability 
of  observance  at  least  once  is  0.75  or  higher,  at  least  seven 
tests  must  be  conducted. 

Suppose  there  were  5  independent  vulnerability  events 
possible  on  each  of  the  10  random  shots,  and  each  event 
had  a  0.2  probability  of  occurring  on  each  of  the  10  shots. 
The  probability  that  any  one  of  the  events  would  occur  at 
least  once  during  the  test  program  is  0.89.  The  probability 
that  all  of  the  events  would  occur  at  least  once  (not  necessar¬ 
ily  on  the  same  shot)  during  the  program  is 

0.895  =  0.56 

Hence,  there  is  a  0.44  probability  that  one  or  more  of  the 
five  kill  modes  will  not  be  observed.  Thus,  the  test  program 
is  essentially  equally  likely  either  to  reveal  ail  or  to  miss  one 
or  more  of  the  aircraft’s  vulnerabilities.  If  each  vulnerability 
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could  occur  on  only  2  of  the  10  test  shots,  the  probability 
that  one  or  more  vulnerabilities  would  not  occur  in  the  10 
tests  is  essentially  one.14 

Synergism  Among  the  Damage  Processes  and  Kill 
Modes.  Live  Fire  Tests  are  also  a  valuable  source  of  vulner¬ 
ability  information  on  the  synergism  among  the  damage  pro¬ 
cesses  and  kill  modes,  albeit  a  limited  source.  Suppose  there 
are  two  kill  modes  I  and  II  that  exhibit  synergism.  Mode  I  has 
a  probability  of  occurrence  of  0. 1 ,  regardless  of  the  occur¬ 
rence  of  mode  II.  Mode  II  has  a  probability  of  occurrence  of 
0. 1  if  the  first  mode  does  not  occur  and  a  0.5  probability  of 
occurrence  if  the  first  mode  does  occur.  On  one  shot,  the  prob¬ 
ability  that  both  modes  occur  is  0.05.  The  probability  that 
only  mode  I  occurs  is  0.05,  the  probability  that  only  mode  II 
occurs  is  0.09,  and  the  probability  that  neither  mode  occurs 
is  0.8 1 .  If  these  modes  were  independent,  and  both  had  a  prob¬ 
ability  of  occurrence  of  0. 1 ,  mode  I  only  occurs  with  a  proba¬ 
bility  of  0.09,  mode  Q  only  occurs  with  a  probability  of  0.09, 
modes  I  and  II  together  occur  with  a  probability  of  0.01,  and 
neither  mode  occurs  with  a  probability  of  0.81.  Thus,  the 
probability  that  neither  mode  occurs  is  the  same  in  both  the 
synergistic  case  and  the  independent  case.  In  the  synergistic 
case,  mode  II  is  more  likely  to  occur  (0.14  vs.  0.10)  and  the 
probability  that  both  modes  occur  together  is  higher  than  in 
the  independent  case  (0.05  vs.  0.01). 

Validation  of  the  Analytical  Model .  The  analytical  models 
are  presently  structured  to  provide  kill  probabilities  and 
vulnerabilities  based  upon  a  selected  kill  category.  The  Live 
Fire  Tests  do  not  directly  provide  the  numerical  data  required 
to  validate  the  model's  predictions;  they  only  provide  infor¬ 
mation  on  the  components  that  were  damaged  or  killed,  the 
occurrence  or  non-occurrence  of  the  kill  modes,  and  any 
cascading  damage.  Thus,  some  compromises  must  be  made 
that  require  additional  analyses  in  order  to  relate  the  empiri¬ 
cal  test  results  to  actual  combat  conditions  and  to  determine 
if  the  test  results  correspond  to  an  actual  kill  of  the  aircraft 
If  enough  test  shots  could  be  made  under  the  identical  set 
of  conditions,  a  statistical  inference  could  be  made  regarding 
the  probability  of  an  aircraft  kill  given  the  test  conditions. 
Unfortunately,  the  number  of  identical  shots  that  can  be 
conducted  is  usually  small,  and  hence  the  confidence  level 
in  the  sample  mean  probability  of  kill  is  low.  Furthermore, 
because  the  models  are  expected  value  models,  and  given 
the  randomness  of  the  empirical  results  from  the  limited 
number  of  tests,  the  likelihood  that  the  test  results  and  the 
model  predictions  are  in  general  agreement  with  respect  to 

the  components  affected  and  their  vulnerability  is  low.  For 

i 

,4These  examples  are  given  to  put  LFT  in  the  same  context  with  the 
analysis/modeling  methodology.  It  is  not  the  purpose  of  LFT  to  find  all 
vulnerabilities. 


example,  suppose  the  model  predicts  a  wing  fuel  tank  will 
explode  when  hit  with  a  probability  of  0.5.  If  each  of  three 
identical  tests  result  in  an  explosion,  does  this  mean  the 
model  is  inaccurate?  The  answer  is  no.15  Thus,  the  Live 
Fire  Tests  can  not  be  used  in  themselves  to  directly  validate 
the  models,  and  they  are  not  conducted  for  that  purpose. 
However,  they  do  provide  valuable  information  on  damage 
processes  that  should  be  modeled. 

Neglect  of  On-Board  Munitions ..  The  Live  Fire  Test  law 
requires  the  aircraft  to  be  full-up  when  tested.  This  means 
that  any  munitions  normally  carried  by  the  aircraft  must  be 
on-board  at  the  time  of  the  test.  This  will  probably  be  done 
in  only  a  very  few  tests;  it  is  not  the  intent  of  the  Live  Fire 
Test  law  to  destroy  aircraft  needlessly.  The  intent  of  the  law 
is  to  obtain  information  on  the  vulnerability  of  the  aircraft 
sufficiently  early  to  allow  any  design  deficiency  to  be  cor¬ 
rected.  Consequently,  it  does  not  seem  reasonable  to  inten¬ 
tionally  create  a  situation  in  which  the  aircraft  could  be 
destroyed  when  essentially  the  same  information  on  the  vul¬ 
nerability  of  the  aircraft  to  the  on-board  munitions  can  be 
obtained  by  using  realistic  off-line  tests  of  sub-scale  models 
of  the  portion  of  the  aircraft  in  the  vicinity  of  the  munitions. 
This  is  particularly  true  for  aircraft  with  internally  stored 
munitions.  Consequently,  inert  surrogates  most  likely  will 
be  used  for  munitions  on-board  the  full-scale  aircraft  in  order 
to  prevent  a  catastrophic  kill.  The  increase  (or  decrease16) 
in  aircraft  vulnerability  due  to  the  presence  of  the  munitions 
can  be  determined  by  relating  the  observed  projectile  and 
fragment  impacts  on  these  surrogates  to  the  vulnerability 
data  on  munition  reactions  to  impacts  obtained  from  the  off¬ 
line  sub-scale  tests.17  It  is  not  necessary  to  load  the  munitions 
on  the  full-scale  aircraft  in  order  to  determine  the  likelihood 
of  an  adverse  munition  reaction  to  a  hit  or  a  fire.  If  a  design 
deficiency  with  respect  to  the  on-board  munitions  is  discov¬ 
ered  and  a  less  vulnerable  design  can  be  developed,  it  can 
be  incorporated  without  the  loss  of  the  test  aircraft 

The  Cost  of  Live  Fire  Testing .  No  complete  Live-Fire 
Testing  program  for  any  aircraft  has  been  carried  out  as  yet. 
Consequently,  the  following  information  is  offered  only  as 
an  indication  of  the  costs  involved  in  full-scale  Live  Fire 
Testing.  If  no  waiver  has  been  given  from  full-scale  testing. 


“Nevertheless,  a  prudent  engineer  would  thoroughly  examine  the  results 
of  the  experiment  and  the  analysis  in  an  attempt  to  determine  whether  the 
sequence  of  three  explosions  was  not  indicative  of  a  higher  probability  of 
occurrence. 

“Some  munitions  may  provide  a  shielding  effect,  such  as  the  bomba 
carried  below  a  wing.  Projectiles  and  fragments  that  do  not  penetrate  a 
nonreacting  bomb  are  prevented  from  impacting  and  damaging  the  wutg. 

l7The  off-line  tests  must  provide  sufficient  information  to  enable  a  valid 
evaluation  of  the  reactions  of  the  munitions  and  the  effects  of  those  rcacoona 
on  the  aircraft  including  lower-order  effects  that  are  not  catastrophic. 
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TABLE  2-1  Live  Fire  Test  Options  for  the  C-17A  (IDA,  1989) 

Option 

Test  Item 

Test  Item 

Cost 

($  million ) 

Testing  Cost 1 
($  million ) 

Total  Cost 
($  million) 

i 

Full-up 

aircraft 

154 

4 

158 

2 

Fuselage  & 
one  wing 

107 

4 

111 

3 

Fuselage 
section  &  one 
wing 

60 

4 

64 

4 

One  wing 

36 

4 

40 

5 

One  wing 
leading  edge 

2 

7 

152 

'This  docs  not  include  the  cost  of  repairing  the  damage  for  the  next  shot  Also,  some  members  of  the  committee  believe  that  $4  million  is  too  low. 
2  An  Air  Force  estimate. 


at  least  one  production  or  preproduction  aircraft  is  required.18 
If  the  test  is  carefully  designed,  and  if  the  target  is  reparable 
up  to  the  last  shot,  one  aircraft  should  be  sufficient  The 
cost  of  the  test  aircraft  is  most  likely  the  major  cost  of  the 
total  program.  A  question  arises  as  to  the  proper  cost  of  the 
aircraft  to  use  when  determining  the  program  cost.  Should 
the  actual  construction  cost  of  the  test  aircraft,  which  might 
be  one  of  the  first  five  or  six  aircraft  built,  be  used?  Should 
the  cost  also  include  the  research  and  development  costs, 
or  should  the  average  flyaway  cost  be  used?  The  particular 
cost  used  will  have  a  major  impact  on  the  perceived  benefit 
of  the  Live  Fire  Test  program.  If  one  aircraft  is  tested  out 
of  a  total  aircraft  buy  of  400,  and  the  average  aircraft  cost 
over  the  buy  is  used,  the  cost  of  the  Live  Fire  Tests  will  be 
less  than  0.3%  of  the  total  program  cost. 

Costs  for  the  C-17A.  The  Institute  for  Defense  Analyses 
(IDA)  has  presented  a  number  of  test  program  options,  is¬ 
sues,  and  costs  for  Live  Fire  Testing  the  C-17A  in  a  draft 
report  (IDA,  1989).  The  report  makes  no  recommendations 
on  which  (if  any)  of  the  options  should  be  selected.  The 
costs  of  five  options  are  given  in  Table  2-L 

At  least  one  production  aircraft  is  required  for  option  1. 
If  the  test  is  carefully  designed,  and  if  the  target  is  reparable 
up  to  the  last  shot,  one  aircraft  should  be  sufficient  The 
aircraft  cost  of  $154  million  is  based  on  a  FY1990  recurring 
flyaway  cost  of  $181  million  with  engineering,  tooling,  and 
avionics  costs  removed.19  On  the  other  hand,  the  FY1997 


"The  use  of  static  test  articles  or  prototypes  for  the  LFTs  should  be 
considered. 

'Hn  the  committee’s  evaluation  of  the  cost  of  full-up,  full-scale  aircraft 
estimates  are  based  on  flyaway  cost  without  avionics.  This  would  no  longer 


and  1998  flyaway  costs  are  $78  million.  Thus,  the  cost  of 
the  test  article  depends  upon  the  method  of  bookkeeping 
used.  The  most  elaborate  test  program  suggested  by  IDA  is 
estimated  to  cost  no  more  than  $4  million  and  involves  100 
small  arms  projectiles,  50  small  AAA  HE  rounds,  and  20 
man-portable  infrared  missile  shots. 

Costs  for  the  RAH -66 A  COMANCHE  Helicopter.  The  cost 
of  Live  Fire  Testing  the  COMANCHE  helicopter  has  been 
estimated  by  a  representative  from  the  Ballistic  Research 
Laboratory  and  coordinated  with  the  Program  Manager 
(PM).  Based  on  projected  production  system  fly-away  costs 
for  COMANCHE,  a  full-up  low-rate  initial  production 
(LRIP)  target  that  is  representative  of  an  operational,  com¬ 
bat-configured  system  will  cost  in  excess  of  $7.5  million. 
Given  the  alternative  (and  planned)  use  of  an  engineering 
test  prototype  aircraft  built  up  to  meet  specific  Live  Fire 
Test  requirements  (a  minimal  configuration),  the  cost  would 
be  less.  However,  it  still  is  a  major  percentage  of  the  LFT&E 
program  expense.  At  least  two  sets  of  target  components/ 
subsystems,  plus  repair  provisions,  would  also  be  needed. 
Steps  can  be  taken  to  attempt  to  minimize  the  risks  to  the 
hardware,  but  these  may  not  always  work.  The  operational 
reutilization  of  the  test  articles  is  unlikely.20 

The  number  of  shots  that  can  be  conducted  on  one  target 
is  highly  variable.  However,  given  ideal  repair  capability 
and  no  catastrophes,  25  or  more  small-caliber  API  shots 


be  a  “full-up”  configuration.  In  fact  recent  testing  on  aircraft  shows  avionics 
provide  excellent  shielding.  In  some  aircraft  avionics  components  are 
themselves  vulnerable. 

*It  should  be  noted  that  all  of  the  test  vehicles  used  in  the  U.S.  Army's 
ABRAMS  Live  Fire  Tests  were  repaired  and  returned  to  service. 


evaluation  of  cost,  effectiveness,  and  deficiencies 
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TABLE  2-2A  Relative  Advantages  and  Disadvantages  of  the  Two  Methodologies:  Type,  Amount,  and  Applications 
of  the  Information 


Analysis/ Modeling 

Live  Fire  Testing 

Sub-scale  Testing 

Full-scale  Testing 

Advantages 

•  The  information  is  available  in  the 
form  of  vulnerable  areas  and 
probabilities  of  kill  given  a  hit. 

•  The  models  can  provide  much 
information  on  the  vulnerability  of  the 
aircraft  over  a  wide  range  of  conditions 
in  which  the  systems  will  be  used,  over 
a  wide  range  of  potential  threat 
weapons,  and  over  a  wide  range  of 
system  design  alternatives. 

•  The  information  can  be  used  to  aid  in 
the  early  design  of  the  aircraft  to 
satisfy  program  requirements,  to  support 
subsequent  campaign/war  game 
assessments,  to  predict  test  outcomes, 
and  to  support  acquisition  decisions. 


Advantages 

•  The  type  of  information  from  sub¬ 
scale  testing  is  in  the  form  of  semi- 
realistic  physical  outcomes,  such  as 
penetration  or  no  penetration,  fire  or  no 
fire,  etc. 

•  The  information  from  sub-scale  tests 
can  be  used  to  aid  the  design  (when  the 
tests  are  conducted  sufficiently  early  in 
the  program),  to  validate  the  design,  to 
provide  data  for  component  data 
bases  and  new  damage  processes  in 
support  of  subsequent  assessments,  and 
to  support  acquisition  decisions. 


Advantages 

•  The  type  of  information  from  full- 
scale  testing  is  in  the  form  of  realistic 
physical  outcomes,  including  cascading 
and  synergistic  damage,  such  as  which 
components  are  damaged,  if  a  fire 
occurred  how  far  it  spread,  etc. 

•  The  information  from  full-scale  tests 
can  be  used  to  aid  the  design  (when  the 
tests  are  conducted  sufficiently  early  in 
the  program),  to  provide  information  on 
new  damage  processes  in  support  of 
subsequent  assessments,  to  satisfy  the 
requirements  of  the  Live  Fire  Test  Law, 
and  to  support  acquisition  decisions. 

•  The  Live  Fire  Tests  that  were 
performed  on  full-scale  armored 
vehicles  provided  data  on  the  types  of 
damage  that  resulted  in  delayed 
catastrophic  damage.  These  data  were 
useful  in  training  the  crew  in  emergency 
exit  procedures.  Similar  data  on  ejection 
procedures  might  come  from  Live-Fire 
Tests  of  aircraft. 

•  Damaged  aircraft  offer  an  opportunity 
for  battle  damage  crews  to  practice 
repairing  realistic  combat  damage. 


Disadvantages 

•  Model  results  reflect  the  assumptions 
on  vulnerability  and  therefore  do  not 
provide  any  new  information  on 
vulnerabilities  resulting  from  new 
materials  and  damage  processes. 


Disadvantages 

•  The  amount  of  information  is 
relatively  small,  is  confined  to  the  tested 
components  and  subsystems,  and  is 
limited  to  the  specific  test  conditions. 

•  The  information  from  sub-scale  tests 
may  not  be  directly  relatable  to  an 
aircraft  kill. 


Disadvantages 

•  The  amount  of  information  is 
relatively  small,  is  confined  to  the  tested 
regions,  and  is  limited  to  the  specific 
test  conditions.  An  unexpected 
catastrophic  reaction  in  one  of  the  early 
tests  can  prevent  the  gathering  of 
additional  information. 

•  The  information  from  full-scale  tests 
may  not  be  directly  related  to  an  aircraft 
kill 

•  Because  of  production-like  aircraft 
must  be  tested  in  the  full-scale  test,  it 
may  not  be  practical  to  correct  design 
deficiencies  discovered  in  the  tests. 

•  Full-scale  tests  will  not  be  used  to 
produce  the  necessary  broad  and 
detailed  data  base  for  supporting  the 
models  because  of  the  lack  of  control 
over  the  test  outcomes  and  the  costs 
involved. 
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TABLE  2-2B  Relative  Advantages  and  Disadvantages  of  the  Two  Methodologies:  Accuracy  of,  and  Level  of 
Confidence  in,  the  Information 

Analysis!  Modeling 

Live  Fire  Testing 

Sub-scale  Testing 

Full-scale  Testing 

Advantages 

Advantages 

Advantages 

•  Models  can  provide  estimates  of  the 
major  vulnerabilities,  particularly  when 
they  are  supported  by  sub-scale  live  fire 
tests. 

•  Sub-scale  tests  can  provide 
vulnerability  data  on  new  materials, 
new  components,  new  construction 
processes,  new  threats,  etc. 

•  Short  of  actual  combat,  full-scale  live 
fire  testing  is  as  close  to  real  data  on  a 
combat-configured  aircraft  as  is  possible 
to  achieve.  It  accounts  for  the  weapons 
effects,  synergism  among  damage 
processes,  and  cascading  damage. 

Disadvantages 

Disadvantages 

Disadvantages 

•  The  models  will  never  be  able  to 
totally  represent  all  of  the  possible 
vulnerability  events  that  can  occur  in 
combat,  such  as  synergism  and 
cascading  damage.  Some  damage 
processes  are  modeled  poorly,  and  some 
are  not  modeled  at  all.  Thus,  the 
estimates  of  vulnerability  do  not  have  a 
high  level  of  confidence,  particularly  for 
new  aircraft. 

•  Current  models  tend  to  underestimate 
vulnerability  because  of  their 
conservative  assumptions  regarding 
weapon  lethality  (e.g.,  they  neglect 
spall,  fuel  ingestion,  and  multiple  hit 
effects). 

•  Sub-scale  tests  can  vary  from  simple 
component  tests  to  realistic  tests  on 
complete  subsystems.  However,  because 
these  tests  cannot  simulate  any 
interaction  between  the  tested  article 
and  the  portions  of  the  aircraft  not 
included,  they  are  not  fully 
representative  of  the  actual  combat 
situation. 

•  Those  weapons  likely  to  be 
encountered  in  combat  that  can  destroy 
the  aircraft  are  not  used  in  such  a 
manner.  They  are  accepted  as 
overmatching  threats. 

•  Altitude  effects  and  the  conditions  of 
flying  are  not  included  in  full-scale 
vulnerability  testing. 

•  Because  the  air  crew  is  not  on  board 
during  an  LFT,  the  possible  effects  of 
the  test  on  the  crew  are  difficult  to 
determine. 

•  There  are  serious  data  gaps  on 
componant  and  subsystem 
vulnerabilities,  such  as  those  identified 
by  the  1991  JTCG/AS  Pw  Workshop. 

plus  four  or  more  small-caliber  HEI  projectile  shots  might 
be  possible.  The  cost  per  shot  is  highly  variable,  ranging 
from  $500  to  $  10,000“,  and  is  a  function  of  the  weapon 
used,  the  target  cbnfiguration,  the  scope  of  the  test  issues, 
the  extent  of  the  instrumentation  and  data  capture,  the 
workup  costs,  etc. 


Advantages  and  Disadvantages  of  Analysis/Modeling 
and  Live  Fire  Testing 

Both  analyst  ^/modeling  and  Live  Fire  Testing  have  ad¬ 
vantages  and  disadvantages  pr  deficiencies  relative  to  one 


-‘Some  committee  members  believe  that  the  cost  of  the  tests  would 
range  from  S3 .000  to  $50,000. 


another.  Furthermore,  there  are  relative  advantages  and  dis¬ 
advantages  of  both  sub-scale  testing  and  full-scale  testing. 
Table  2-2  presents  a  comparison  of  the  methodologies  for 
each  of  the  three  information  attributes  of  type,  amount,  and 
applications;  accuracy  or  level  of  confidence;  and  cost. 

Conclusion 

The  committee  concludes  that  the  combination  of  analyt¬ 
ical  models,  supported  by  live  fire  tests  on  components 
and  subsystems,  and  the  sub-scale  and  full-scale  Live  Fire 
Tests  are  mutually  compatible  in  the  vulnerability  analysis, 
evaluation,  and  design  of  aircraft  They  complement  each 
other,  and  the  whole  is  superior  to  the  sum  of  the  parts. 
More  work  is  needed  to  unify  these  approaches  in  order 
to  obtain  the  maximum  benefit  The  aircraft  vulnerability 
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TABLE  2-2C  Relative  Advantages  and  Disadvantages  of  the  Two  Methodologies:  Cost 

Analysis/ Mode  ling 

Live  Fire  Testing 

Sub- scale  Testing 

Full-scale  Testing 

Advantages 

•  The  cost  of  analysis  is  significantly 
less  than  the  cost  of  Live  Fire  Testing. 

Advantages 

•  Sub-scale  tests  are  considerably  less 
expensive  than  full-scale  tests. 

Advantages 

•  Live  Fire  Tests  are  a  small 
percentage  of  the  total  program  costs.  If 
only  one  aircraft  is  tested  out  of  a  total 
aircraft  buy  of  400,  the  cost  of  the  Live 

Fire  Test  program  is  less  than  0.3%  of 
the  total  program  cost. 

•  The  use  of  static  test  articles  or 
prototypes  can  significantly  reduce  the 
cost  of  the  tests. 

Disadvantages  . 

•  Although  the  actual  cost  of 
conducting  an  analysis  is  relatively  low, 
there  are  considerable  sub-scale  test 
costs  required  to  build  up  the  essential 
vulnerability  data  base. 

Disadvantages 

•  Although  sub-scale  tests  are  less 
costly  than  full-scale  tests,  a 
comprehensive  sub-scale  test  program  in 
support  of  a  particular  aircraft 
acquisition  program  is  not  inexpensive, 
and  the  program  manager  may  not  be 
inclined  to  spend  the  required  funds. 

•  The  cost  of  sub-scale  testing  may 
preclude  testing  for  basic 
phenomenological  data  on  damage 
processes  and  kill  modes. 

Disadvantages 

•  The  cost  of  full-scale  testing  is 
primarily  influenced  by  the  cost  of  the 
test  aircraft.  Thus,  the  cost  of  testing  a 
full-scale  aircraft  is  considerably  more 
than  the  cost  of  analyses  or  sub-scale 
testing  when  the  aircraft  was  procured 
only  for  that  purpose  and  cannot  be 
repaired  and  put  into  service. 

community,  based  on  its  plans  for  the  Live  Fire  Tests, 
seems  to  appreciate  the  need  to  integrate  these  approaches, 
having  witnessed  the  success  in  using  the  data  from  the 
Live  Fire  Tests  on  the  Abrams  tank  and  the  Bradley 
Fighting  Vehicle  to  improve  both  the  analytical  methodol¬ 
ogy  and  the  vehicle  designs. 
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The  OSD  Live  Fire  Test 
and  Evaluation  Program 


The  Live  Fire  Test  law  passed  in  1987  stipulates  that 
realistic  survivability  testing  be  conducted  on  covered  sys¬ 
tems.  Realistic  survivability  testing  is  defined  as  “ testing 
for  vulnerability  of  (he  system  in  combat  by  firing  munitions 
likely  to  be  encountered  in  combat  (or  munitions  with  a 
capability  similar  to  such  munitions)  at  the  system  config¬ 
ured  for  combat ,  with  the  primary  emphasis  on  testing  vul¬ 
nerability  with  respect  to  potential  user  casualties  and  taking 
into  equal  consideration  the  susceptibility  to  attack  and 
combat  performance  of  the  system9  (U.S.  Congress,  1986— 
1989).  Systems  covered  by  the  law  are  vehicles,  weapon 
platforms,  or  conventional  weapon  systems  that  include  fea¬ 
tures  designed  to  provide  some  degree  of  protection  to  users 
in  combat  and  are  major  systems  within  the  meaning  of  that 
term  in  section  2303(5)  of  title  10,  i.e..  Acquisition  Category 
I  and  II  systems.  The  Secretary  of  Defense  may  waive  the 
requirement  for  Live  Fire  Testing  of  candidate  systems  if, 
before  the  system  enters  full-scale  development,  the  Secre¬ 
tary  “ certifies  to  the  Congress  that  such  testing  would  be 
unreasonably  expensive  and  impractical 99  (U.S.  Congress, 
1986-1989).  The  request  for  a  waiver  will  be  prepared  by 
the  Service  and  submitted  to  the  Secretary  through  the  appro¬ 
priate  chain  of  command.  If  the  waiver  is  granted,  the  Secre¬ 
tary  shall  include  with  any  such  certification  a  report  ex¬ 
plaining  how  the  Secretary  plans  to  evaluate  the  survivability 
or  the  lethality  of  the  system  or  program  and  assessing 


possible  alternatives  to  realistic  survivability  testing  of  the 
system  or  program  (U.S.  Congress,  1986-1989). 

Personnel  from  the  Live  Fire  Test  Office  and  from 
the  Deputy  Director,  Defense  Research  and  Engineer¬ 
ing  (Test  and  Evaluation/Tactical  Warfare  Program) 
[DDDR&E(T&E)/TWP]  presented  the  Office  of  the  Secre¬ 
tary  of  Defense  (OSD)  Live  Fire  Test  and  Evaluation 
(LFT&E)  program  to  the  committee  on  July  24,  1991,  and 
the  Director  of  the  Live  Fire  Office  made  a  second  presenta¬ 
tion  to  the  committee  on  January  15, 1992.  Personnel  from 
the  Institute  for  Defense  Analyses  gave  presentations  to  the 
committee  on  September  26,  1991. 


Oversight  and  Published  Guidance  for  LFT&E .  When 
the  Live  Fire  Test  law  was  passed  in  1987,  the  position  of 
Director,  Live  Fire  Testing,  was  established  under  the  Office 
of  the  Director  of  Defense  Research  and  Engineering  (Test 
and  Evaluation)  and  was  given  the  responsibility  for  imple¬ 
menting  the  Live  Fire  Test  (LFT)  legislation,  developing 
and  issuing  guidelines  for  test  plan  preparation,  reviewing 
and  approving  Service-prepared  plans,  performing  indepen¬ 
dent  assessments  of  the  test  results,  providing  liaison  with 
Congress  on  the  test  results,  and  working  with  the  Services 
and  the  Department  of  Defense  (DoD)  in  improving  LFT 
methodology  and  instrumentation,  and  in  general,  ensuring 


38 


OSD  LIVE  FIRE  TEST  AND  EVALUATION  PROGRAM 


39 


that  conduct  of  Live  Fire  Testing  is  within  the  spirit  of 
congressional  intent  (0‘Bryon,  1987).  A  series  of  written 
documents  has  been  prepared  by  OSD  to  provide  guidance 
regarding  details  not  specifically  contained  in  the  law. 

The  first  set  of  guidelines  intended  to  implement  the 
congressionally  mandated  Live  Fire  Test  program  were  is¬ 
sued  on  May  7,  1987  (DDDR&E,  1987).  These  guidelines 
supplemented  DoD  Directive  5000.3-M-l,  ‘Test  and  Evalua¬ 
tion  Master  Plan  Guidelines/’  October  1986,  in  areas  per¬ 
taining  to  Live  Fire  Testing.  They  also  defined  the  LFT&E 
plan  requirements.  According  to  these  guidelines,  “the  essen¬ 
tial  feature  of  Live  Fire  Testing  is  that  threat  munitions  are 
fired  against  a  major  U.S.  system  configured  for  combat  to 
test  its  vulnerability,  and/or  that  a  major  U.S.  major  munition 
or  missile  is  fired  against  a  threat  target  configured  for  com¬ 
bat  to  test  the  lethality  of  the  munition  or  missile.  Each 
(Dive  (F)ire  (T)est  plan  must  include  testing  of  complete 
systems.  However,  a  limited  set  of  live  fire  tests  may  involve 
production  components  configured  as  a  subsystem  prior  to 
full  up  testing.  In  such  a  case  the  components  must  be  tested 
in  the  context  of  the  complete  system  in  that  the  test  issues 
must  be  specific  to  both  a  threat  system  and  a  target  system. 
In  addition,  at  least  a  preliminary  decision  shall  have  been 
made  to  configure  the  production  system  with  the  compo¬ 
nents  tested.  Thus,  (L)ive  (F)ire  (Testing  is  not  synonymous 
with  traditional  research  and  development  vulnerability/le- 
thality  testing”  (DDDR&E,  1987). 

The  1988  Test  and  Evaluation  Committee  (LFT &E)  Guide¬ 
lines,  The  1987  LFT&E  Guidelines  were  superseded  by 
the  Live  Fire  Test  and  Evaluation  Guidelines  issued  June  1, 
1988,  by  the  Test  and  Evaluation  Committee  (TEC),  Office 
of  the  Secretary  of  Defense,  in  a  memorandum  to  the  Assis¬ 
tant  Secretary  of  the  Army  (Research  Development  and  Ac¬ 
quisition),  the  Assistant  Secretary  of  the  Navy  (Research 
Development  and  Acquisition),  and  the  Assistant  Secretary 
of  the  Air  Force  (Acquisition)  (OSD,  1988).  The  TEC  memo¬ 
randum  stated  that  the  enclosed  guidelines  implemented  the 
congressionally-mandated  Live  Fire  Test  program  within  the 
DOD.  The  guidelines  had  been  reviewed  and  coordinated 
by  the  individual  Service  deputies  responsible  for  test  and 
evaluation.  According  to  the  memorandum,  “the  LFT  pro¬ 
gram  is  a  highly  visible  and  sometimes  misunderstood  pro¬ 
gram.  These  guidelines  should  eliminate  much  of  that  misun¬ 
derstanding ”  (OSD,  1988).  The  guidelines  describe  the 
objectives  of  LFT&E,  the  scope  of  the  guidelines,  the  imple¬ 
mentation  of  the  program,  and  the  responsibilities  of  the 
DoD  staff  and  the  Services  relative  to  LFT&E. 

i 

» 

The  1989  LFT&E  Planning  Guide.  A  Live  Fire  Test  and 
Evaluation  (LFT &E)  Planning  Guide  was  issued  in  June 
1989  by  the  Director,  Live  Fire  Testing  (DDDR&E,  1989). 


This  Planning  Guide  is  the  current  primary  source  of  infor¬ 
mation  on  LFT&E  provided  by  the  LFT  office  to  the  Services 
and  industry.  The  Planning  Guide  was  intended  to  provide 
a  good  foundation  for  understanding  the  LFT&E  program, 
and  to  be  useful  to  system  program  offices  and  test  agencies 
responsible  for  the  testing  and  evaluation  of  systems  identi¬ 
fied  as  Live  Fire  candidates;  nothing  in  the  Planning  Guide 
was  intended  to  be  inconsistent  with  the  Live  Fire  legislation 
or  any  related  DoD  directives.  The  Planning  Guide  contains 
the  TEC  LFT&E  guidelines  of  1988;  the  Live  Fire  Testing 
legislation;  and  guidance  concerning  the  definition  of  critical 
issues  for  Live  Fire  testing,  the  development  of  a  strategy 
for  LFT&E,  the  preparation  of  a  detailed  LFT&E  plan,  and 
the  integration  process  involving  contracting,  budgeting,  and 
scheduling.  According  to  the  Planning  Guide,  the  intent  of 
LFT&E  (with  regard  to  vulnerability)  is  “to  assure  that  battle 
damage  tolerance  and  damage  control  of  our  crew-carrying 
combat  systems  to  actual  threat  weapons  is  known  and  ac¬ 
ceptable.  . 

The  LFT  Program  Approach 

In  concert,  the  LFT&E  Guidelines  of  1988  and  the 
LFT&E  Planning  Guide  of  1989  require  the  services  to 
take  an  ordered  approach  to  LFT&E  on  all  covered  aircraft 
systems.  This  approach  includes  the  following  steps: 

1.  identification  of  the  critical  issues; 

2.  development  of  an  LFT&E  strategy,  which  must  be 
an  integral  part  of  the  Test  and  Evaluation  Master  Plan 
(TEMP)  and  is  subject  to  OSD  review  and  approval; 

3.  integration  of  this  strategy  into  the  covered  system 
program  plan,  budget,  and  schedule; 

4.  development  of  a  detailed  LFT&E  test  plan  that  fully 
describes  the  tests  to  be  conducted  and  criteria  for  measuring 
test  results  and  is  subject  to  OSD  review  and  comment; 

5.  conduct  of  the  tests;  and 

6.  generation  of  a  detailed  test  report  subject  to  OSD 
review  for  submission  to  the  Secretary  of  Defense  and  the 
congressional  committees. 

The  identification  of  the  critical  issues  is  required  to  form 
a  foundation  upon  which  a  strategy  for  the  Live  Fire  Test 
program  can  be  developed  for  each  system.  To  fully  under¬ 
stand  the  system  and  the  threat,  information  that  fully  de¬ 
scribes  the  requirements  for  the  system,  the  environment  in 
which  it  must  operate,  and  concepts  for  operation  of  the 
system  must  be  gathered.  Vulnerability  analyses  must  then 
be  conducted  to  identify  potential  weaknesses  in  the  system 
and  to  obtain  first-order  assessments  of  the  ability  of  the 
system  to  meet  its  operational  requirements.  Given  this  infor¬ 
mation,  vital  vulnerability  concerns  can  be  identified  and 
distilled  into  the  critical  Live  Fire  Test  issues.  Depending 
upon  the  system,  there  are  several  sources  for  Live  Fire  Test 


i 


40 


VULNERABILITY  ASSESSMENT  OF  AIRCRAFT 


issues.  For  example,  if  the  system  is  designed  to  replace  an 
existing  system,  a  Live  Fire  Test  issue  would  be.  How  does 
the  vulnerability  of  the  new  system  compare  with  that  of 
the  existing  system? 

The  next  step  in  the  planning  process  is  the  development 
of  a  strategy  for  conduct  of  the  Live  Fire  Test  program. 
This  strategy  shall  include  the  establishment  of  measures  of 
evaluation,  procedures  for  the  evaluation  of  the  results,  the 
data  requirements,  and  the  test  objectives.  This  strategy  shall 
be  a  part  of  the  Test  and  Evaluation  Master  Plan. 

Provision  must  next  be  made  for  the  integration  of  the 
Live  Fire  Test  strategy  into  the  program  plan  for  the  develop¬ 
ment  of  the  system.  The  Program  Manager  will  ensure  that 
the  overall  program  plan,  budget,  and  schedule  provide  ade¬ 
quate  resources  to  assure  a  successful  Live  Fire  Test 
program. 

A  detailed  Live  Fire  Test  plan  is  required  that  fully  de¬ 
scribes  the  tests  to  be  conducted  and  the  criteria  for  evaluat¬ 
ing  test  results.  This  test  plan  shall  be  in  sufficient  detail  to 
ensure  that  the  tests  will  satisfy  the  strategy  objectives.  The 
test  plan  is  subject  to  review  and  comment  by  the  Director, 
Live  Fire  Test 

The  conduct  of  the  tests  is  to  be  witnessed  by  the  Live 
Fire  Test  Office,  which  does  an  independent  evaluation  of 
the  results.  The  Services  prepare  a  test  report  that  is  reviewed 
by  the  Live  Fire  Test  Office.  The  test  report  is  then  forwarded 
to  the  Secretary  of  Defense  with  the  independent  OSD  evalu¬ 
ation  of  the  results,  and  from  there  it  is  sent  to  the  interested 
congressional  committees. 

The  1988  LFT&E  Guidelines  and  the  1989  LFT&E  Plan¬ 
ning  Guide  are  the  current  written  guidance  provided  to  the 
Services  and  their  Program  Managers  to  assist  them  in  the 
planning  and  conduct  of  a  Live  Fire  Test  program  that  is 
guaranteed  to  satisfy  the  requirements  of  the  OSD  and, 
through  the  law,  the  Congress.  In  addition  to  this  written 
material,  the  staff  of  the  Live  Fire  Test  office  is  available 
for  regular  consultation  with  the  program  offices  in  the  prep¬ 
aration  of  the  strategy  for  the  TEMP  and  of  the  detailed  test 
plan.  Other  methods  used  by  the  LFT  Office  to  communicate 
the  LFT  program  to  the  testing  and  evaluation  (T&E)  and 
acquisition  communities  include  full  membership  for  the 
LFT  Director  on  the  Defense  Acquisition  Board  Committees, 
periodic  one-on-one  discussions  with  the  Services*  Live  Fire 
Test  gate  keepers,  introduction  of  the  program  into  the  curric¬ 
ulum  of  the  Defense  System  Management  College,  prepara¬ 
tion  of  a  video  containing  an  overview  of  the  program,  public 
testimony  before  congressional  defense  subcommittees, 
sponsorship  of  six  Live  Fire  Lessons  Learned  Workshops, 
participation  in  more  than  25  T&E  symposia,  more  than  40 
on-site  overview  question-and-answer  presentations  to  the 
major  defense  contractors,  and  publication  in  the  open  litera¬ 
ture  of  more  than  30  articles  on  Live  Fire  Testing  (O’Bryon, 
1991). 


Controversy  Regarding  the  Definition  of  “Realistic 
Survivability  Testing” 

The  committee  is  concerned  that  the  written  guidance 
does  not  provide  sufficient  detail,  particularly  with  respect 
to  the  full-scale  tests,  to  ensure  that  the  Program  Manager 
can  satisfy  the  requirements  of  the  OSD  policy  and  the  law 
and  also  design  a  cost-effective  test  plan  that  will  ensure 
system  requirements  are  satisfied.  Furthermore,  the  commit¬ 
tee  is  concerned  about  the  official  status,  or  lack  thereof,  of 
the  LFT&E  Guidelines  and  Planning  Guide.1 

In  addition  to  the  questions  regarding  the  level  of  detail 
and  the  current  status  of  the  guidance,  there  is  a  question 
regarding  the  interpretation  of  specific  mandates  contained 
in  the  Live  Fire  Test  law.  A  major  issue  apparently  exists 
between  the  Services  and  OSD,  and  indeed  within  OSD 
itself,  regarding  the  definition  and  meaning  of  “realistic  sur¬ 
vivability  testing”  given  in  the  law.  The  definition  has  been 
interpreted  by  the  1988  Live  Fire  Test  and  Evaluation  Guide¬ 
lines  as  follows: 

a.  Live  Fire  Test:  A  test  event  within  an  overall  LFT&E 
program  which  involves  the  firing  of  actual  munitions  at 
target  components,  target  subsystems,  target  subassemblies, 
andior  full-scale  targets  to  examine  personnel  casualty,  vul¬ 
nerability,  and/or  lethality  issues . 

b.  Full-up  Testing:  Firings  against  a  full-scale  target  con¬ 
taining  all  the  dangerous  materials  (e.g.,  ammunition,  fuel, 
hydraulic  fluids,  etc.),  system  parts  (e.g.,  electrical  lines  with 
operating  voltages  and  currents  applied,  hydraulic  lines 
containing  appropriate  fluids  at  operating  pressures,  etc.), 
and  stowage  items  normally  found  on  that  target  when  oper¬ 
ating  in  combat.  Full-up  testing  includes  firings  against  full- 
up  components,  full-up  subsystems,  full-up  subassemblies, 
or  full-up  systems.  The  term  " Full-up  Testing"  is  synony¬ 
mous  with  “ realistic  survivability  testing"  or  u realistic  le¬ 
thality  testing"  as  defined  in  the  legislation  covering  Live 
Fire  Testing. 

The  phrasing  of  these  definitions  has  been  interpreted  by 
some  to  imply  that  full-scale,  fuil-up  Live  Fire  Tests  do  not 
have  to  be  conducted  (i.e.,  OSD  policy  is  satisfied  by  Live 
Fire  Tests  only  on  sub-scale  targets,  such  as  major  portions 
or  subassemblies  of  an  aircraft).  The  essence  of  the  problem 
is  the  phrase  in  the  law  that  states  that  realistic  survivability 
testing  means  “testing  for  vulnerability  of  the  system  by.” 
What  is  the  “system?”  Does  testing  only  one  major  subas¬ 
sembly  constitute  testing  for  vulnerability  of  the  system?  If 
it  does,  as  some  believe,  then  no  waiver  from  the  mandated 
system  Live  Fire  Testing  is  necessary. 


’If  action  is  taken  to  assign  official  status  to  these  documents,  they  should 
first  be  modified  to  ensure  that  there  is  no  ambiguity  in  the  requirements 
for  full-scale  testing  (see  discussion  below).  One  briefer  indicated  to  the 
committee  that  guidelines  were  good  for  only  one  yearand,  beyond  that, 
the  “official  status**  would  not  be  valid. 
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Although  these  definitions  in  the  LFT&E  Guidelines  are 
apparently  the  official  guidance  on  realistic  survivability 
testing  provided  to  the  Services’  Program  Managers,  the 
Live  Fire  Test  Office  does  not  fully  accept  them.  It  believes 
the  Guidelines  make  too  liberal  an  interpretation  of  the  intent 
of  the  Congress.  It  believes  that  the  law  requires  the  testing 
of  complete  systems,  not  just  sub-systems  or  mock-ups,  fully 
configured  for  combat.  However,  it  appears  to  recognize 
that  this  can  be  unreasonably  expensive  and  impractical,  and 
that  circumstances  exist  in  which  the  testing  of  only  sub¬ 
scale  targets  is  justified.  In  those  situations,  it  believes  a 
waiver  should  be  requested. 

The  committee  believes,  as  stated  in  Chapter  1,  that  the 
Live  Fire  Test  law  requires  that  full-scale,  full-up  tests  be 
conducted  on  covered  systems,  unless  a  waiver  is  granted. 
Furthermore,  it  believes  that  the  definitions  in  the  1988 
Guidelines  and  the  guidance  given  in  the  1989  Planning 
Guide  are  not  sufficiently  clear  as  to  the  law’s  requirement 
that  full-scale,  full-up  testing  must  be  conducted  As  a  conse¬ 
quence  of  this  misunderstanding,  the  Services  have  pro¬ 
ceeded  with  sub-scale  Live  Fire  Test  programs  on  several 
weapon  systems  without  making  a  provision  for  testing  a 
complete  system  and  without  asking  for  a  waiver  because 
of  the  belief  that  no  full-scale,  full-up  testing  was  required 
if  early  tests  on  sub-scale  targets  showed  no  design  weak¬ 
nesses.2 

Conclusion 

The  committee  believes  that  the  Services,  when  they 
consider  that  full-scale,  full-up  Live  Fire  Tests  are  unreason¬ 
ably  expensive  and  impractical,  should  ask  for  a  waiver. 
There  should  be  no  stigma  attached  to  a  waiver  if  a  strong 
case  can  be  made  for  one.  According  to  the  law,  “ the  Secre¬ 
tary  of  Defense  may  waive  the  application  of  the  survivability 
and  lethality  tests  of  this  section  to  a  covered  system ,  muni¬ 


: These  programs  are  examined  in  Chapter  4. 


tions  program:  or  missile  program  if  the  Secretary,  before 
the  system  enters  full-scale  engineering  development,  certi¬ 
fies  to  C  ongress  that  live-fire  testing  of  such  system  or  pro¬ 
gram  would  be  unreasonably  expensive  and  impracticar 
(U-S.  Congress,  1986-1989).  If  a  waiver  is  granted,  the 
law  states  that  "the  Secretary  shall  include  with  arty  such 
certification  a  report  explaining  how  the  Secretary  plans  to 
evaluate  the  survivability  or  the  lethality  of  the  system  or 
program  and  assessing  possible  alternatives  to  realistic  sur¬ 
vivability  testing  of  the  system  or  program ”  (U.S.  Congress, 
1986-1989).  Thus,  requesting  and  receiving  a  waiver  from 
the  requirement  for  a  full-scale,  full -up  test  program  do  not 
eliminate  the  requirement  for  vulnerability  assessment;  only 
for  the  full-scale,  full-up  testing  portion  of  the  assessment. 
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Live  Fire  Test  Programs  of  the 
Three  Services,  and  Views  of  the 
Test  Community  and  Industry 


The  Army,  Navy,  and  the  Air  Force  have  developed 
general  Live  Fire  Test  and  Evaluation  (LFT&E)  policies  in 
response  to  the  passage  of  the  1987  Live  Fire  Test  (LFT) 
law,  and  these  policies  are  reflected  in  the  LFT&E  programs 
for  the  RAH -66  (Army),  the  V-22  and  A- 12  (Navy),  and  the 
F-22  and  C-17  (Air  Force).  These  policies  and  the  individual 
aircraft  LFT&E  programs  were  presented  to  the  committee 
on  September  26-27, 1991,  and  are  summarized  below.  Also 
contained  in  this  chapter  are  the  summaries  of  the  presenta¬ 
tions  given  to  the  committee  on  July  25,  1991  by  the  test 
community  from  each  Service.  The  presenters  represented 
the  offices  of  U.S.  Air  Force,  Office  of  the  Director,  Test 
&  Evaluation;  U.S.  Army,  Office  of  the  Director,  U.S.  Army 
Test  and  Evaluation  Management  Agency;  U.S.  Navy,  Office 
of  Test  and  Evaluation  and  Technology  Requirements.  The 
highlights  of  all  of  these  presentations  are  given  below. 

The  committee  believes  it  is  important  to  point  out  before 
proceeding  to  the  presentations  that,  except  for  the  RAH- 
66,  all  of  the  programs  described  below  were  well  under 
way  when  the  law  was  passed  and  the  C-17  development 
had  proceeded  beyond  the  deadline  for  the  application  of  a 
waiver  Congress  had  given  no  transitional  guidance  for 
these  programs.  Furthermore,  because  the  Services  had  only 
recently  begun  to  test  full-scale,  full-up  aircraft  in  the  Joint 
Live  Fire  (JLF)  program,  they  lacked  adequate  test  facilities 
for  conducting  some  of  the  tests;  and  they  did  not  have  any 
long-term  experience  in  preparing  the  test  plans  for,  and 
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conducting,  such  tests.1  Finally,  because  of  the  confusion 
over  the  requirements  of  the  law,  no  waivers  were  requested 
for  any  program  because  of  the  belief  within  the  Services 
that  the  law  was  satisfied  with  less-than-fuil-scale,  full-up 
tests  and  that  a  stigma  would  be  attached  to  any  waiver. 
This  situation  has  complicated  the  preparation  of  a  thorough 
LFT&E  program  that  satisfies  the  letter  of  the  law  as  well 
as  its  intent 

The  Army  LFT&E  Programs 

The  Army  gave  two  presentations  on  its  Live  Fire  Testing 
program.  One,  given  by  a  representative  from  the  U.S.  Army 
Test  and  Evaluation  Management  Agency,  was  entitled 
“Army  Philosophy  and  Policy  on  Live  Fire  Testing.”  The 
other,  entitled  “RAH-66  COMANCHE  Ballistic  Vulnerabil¬ 
ity  DT&E  Program  Briefing,”  was  given  by  a  representative 
from  the  COMANCHE  Program  Manager's  office.  The  pre¬ 
sentations  are  summarized  below. 

Army  Philosophy  and  Policy  on  Live  Fire  Testing.  Ac  ¬ 
cording  to  the  presenter,  the  objectives  of  the  Army  Live 


‘The  1987  General  Accounting  Office  study  presented  in  Appendix  B 
addressed  the  issues  of  the  status  of  the  JLF  test  program,  the  methodological 
quality  of  the  testing  and  evaluation  process,  and  how  to  improve  live  fire 
testing. 
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Fire  test  and  evaluation  (T&E)  program  are  to  demonstrate 
the  ability  of  a  system  to  provide  survivability,  to  provide 
insight  into  the  principal  damage  mechanisms  and  kill 
modes,  and  to  provide  insight  into  techniques  for  reducing 
personnel  casualties  and  enhancing  system  survivability.  The 
Army  policy  reiterates  the  legislative  language.  A  draft  pol¬ 
icy  document  AR  73-XX,  “Test  and  Evaluation  Policy/*  was 
released  for  interim  use  in  December  1990.  According  to 
that  document,  LFT&E  will  be  integrated  into  the  overall 
T&E  program  strategy.  The  Live  Fire  Tests  are  part  of 
technical  testing,  and  the  scope  of  the  program  will  build  on 
early  testing  of  components  and  on  modeling.  The  LFT&E 
strategy  will  be  contained  in  the  Test  and  Evaluation  Master 
Plan  (TEMP).  A  draft  document  Department  of  the  Army 
Pamphlet  (DA  PAM)  73-XX,  “Test  and  Evaluation  Proce¬ 
dures  Guidelines/’  is  being  prepared.  This  document  will 
incorporate  the  document  “An  Army  Guide  to  Live  Fire 
Test  and  Evaluation/’  August  1990. 

According  to  the  presenter,  the  Army  LFT&E  strategy 
involves  a  complementary  testing  and  modeling  effort.  Mod¬ 
eling  is  an  essential  part  of  the  strategy  for  determining 
system  vulnerability  since  it  is  impractical  to  test  the  total 
spectrum  of  weapon/aircraft  interactions,  and  testing  is  re¬ 
quired  to  provide  the  necessary  data  to  develop  the  vulnera¬ 
bility  model  algorithms.  The  LFT&E  strategy  follows  a 
“building-block”  approach  consisting  of  component  testing; 
fuli-up  subsystem  testing;  and  full-scale,  full-up  system  test¬ 
ing.  The  emphasis  of  the  testing  program  is  on  front-end 
testing  of  the  components  and  subsystems  to  ensure  that  the 
performance  of  these  tested  articles  is  understood.  Limited 
full-scale,  full-up  testing  is  conducted  to  confirm  this  under¬ 
standing. 


Live  Fire  Testing  of  the  RAH-66  COMANCHE  Helicop¬ 
ter.  The  Program/Ballistics  Detailed  Schedule  for  the 
COMANCHE  (presented  to  the  committee  in  September 
1991)  is  shown  in  Figure  4-1.2  The  ballistic  assessment  tests 
began  in  April  1991  and  ended  in  November  1991.  A  three- 
and-half  year  period  of  ballistic  verification  and  demonstra¬ 
tion  tests  and  Live  Fire  Tests  begins  in  March  1995  and 
ends  in  September  1998. 

The  presenter  identified  three  major  ballistic  vulnerability 
issues  that  could  result  from  expected  threat  encounters. 
The  first  issue  deals  with  the  hazards  to  the  COMANCHE 
aircrew,  the  second  issue  deals  with  the  vulnerabilities  of 


2In  early  1992.  the  Array  restructured  the  COMANCHE  program, 
stretching  out  the  prototype  phase  by  two  years  through  the  summer  of 
1997.  Plans  for  completing  the  development  and  going  into  production 
after  1997  have  been  dropped,  and  the  survivability  and  live  fire  tests  have 
been  deferred.  Apparently,  this  is  part  of  the  Defense  Department’s  plans 
to  emphasize  prototype  research  and  development  of  new  aircraft  (Bond, 
1992). 


the  flight  critical  subsystems,  and  the  third  deals  with  the 
mission  essential' subsystems.  Accordingly,  the  objectives 
of  the  development  test  and  evaluation  (DT&E)  program 
for  ballistic  vulnerability  are  to  verify  adequate  aircrew  and 
aircraft  protection,  to  reduce  the  vulnerability  of  the  flight 
critical  components,  and  to  reduce  the  vulnerability  of  the 
mission  essential  components.  All  subsystems  are  to  be  eval¬ 
uated  in  the  step-by-step  test  program,  starting  from  section/ 
coupon  testing,  proceeding  through  static/dynamic  compo¬ 
nent  testing  and  full-up  component  and  subsystem  testing, 
and  finishing  with  fuli-up  complete  system  Live  Fire  Testing. 
The  early  tests  are  design  support  tests,  and  the  later  tests 
are  to  establish  specification  compliance.  Two  examples  of’ 
design  support  tests  using  coupons  or  engineering  mock- 
ups  are  the  hydraulic  ram  tests  on  fuel  tank  panels  and  the 
ballistic  penetration  tests  on  composite  panels.  Two  of  the 
test  articles  to  be  examined  for  specification  compliance  are 
the  flight  control  linkage  and  the  anti-torque  system  drive 
shaft.  Most  of  the  26  tests  scheduled  will  use  the  12.7- 
millimeter  API;  five  tests  are  scheduled  for  a  larger-caliber 
projectile.  The  Milestone  II  exit  criteria  tests  are  to  demon¬ 
strate  the  damage  tolerance  potential  of  selected  critical  com¬ 
ponents  to  the  specified  design  threats.  Some  of  the  test 
articles  are  the  main  rotor  hub,  main  rotor  blade,  fantail 
assembly,  and  the  T800  engine.  The  focus  of  these  tests  is 
to  verify  the  ability  of  the  helicopter  to  fly  for  30  minutes 
after  a  ballistic  impact 

According  to  the  presenter  from  the  COMANCHE  Pro¬ 
gram  Office,  the  objective  of  the  dedicated  Live  Fire  Test 
program  is  to  evaluate  the  vulnerability  of  the  full-up  produc¬ 
tion  configuration  components,  subsystems,  and  if  necessary, 
a  representative  COMANCHE  air  vehicle.  The  focus  of  these 
tests  will  be  on  postdamage  tolerance  and  the  evaluation  of 
synergistic  effects.  According  to  the  presenter,  the  LFT&E 
program  uses  the  approved  threats;  schedules  the  tests  in  a 
sequential  “building-block”  approach;  tests  full-up  compo¬ 
nents  and  subsystems;  will  test  a  full-scale,  full-up  system, 
if  necessary;  and  is  in  compliance  with  the  law. 


Committee  Comments .  Several  aspects  of  the  COMAN¬ 
CHE  LFT&E  program  are  of  concern  to  the  committee. 
First,  the  schedule  shows  a  Live  Fire  Test  program  being 
completed  just  before  going  into  full-rate  production  but 
after  72  aircraft  have  been  bought  in  low-rate  production 
contract  awards,  which  is  approximately  15%  of  the  planned 
buy.  It  may  not  be  possible  to  retrofit  the  changes  required 
to  correct  any  design  deficiencies  discovered  in  the  LFT&E 
program  into  these  aircraft  Furthermore,  if  approval  is  given 
to  go  into  full  production,  another  96  aircraft  will  be  pro¬ 
duced  in  the  next  year  while  the  design  changes  are  being 
prepared.  This  is  contrary  to  the  LFT  law’s  requirement 
that  testing  shall  be  carried  out  sufficiendy  early  in  the 
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FIGURE  4-1  Program/ballistics  detailed  schedule  for  the  COMANCHE. 


development  phase  of  the  system  to  allow  any  observed 
design  deficiency  to  be  corrected  before  proceeding  beyond 
low-rate  initial  production  (LREP).  There  are  two  ways  this 
situation  could  be  improved.  One  is  that  the  demonstration 
and  validation  (DEM/VAL)  prototypes  will  have  been  flying 
for  several  years  before  the  first  of  the  low-rate  aircraft  will 
have  been  contracted  for,  and  one  of  these  could  be  used 
for  Live  Fire  Testing.  The  other  way  is  to  slow  down  the 
production  rate. 

Second,  the  briefer  was  vague  on  the  actual  testing  of  a 
full-scale,  full-up  aircraft  and,  when  questioned,  said  a  full- 
scale  aircraft  was  not  expected  to  be  needed.  On  the  other 
hand,  his  summary  says  the  program  will  operate  in  compli¬ 
ance  with  the  current  Live  Fire  Test  law.  The  presenter  from 
the  Army  Test  and  Evaluation  Agency  left  no  doubt  of  the 
Army’s  position;  it  will  do  full-scale,  full-up  testing. 

The  COMANCHE  briefer  mentioned  the  on-board  muni¬ 
tions,  but  did  not  dwell  on  its  contribution  to  the  vulnerable 


area  of  the  aircraft  There  was  little  discussion  of  what 
the  internal  stowage  of  ordnance  might  do  to  vulnerability, 
whether  this  would  be  addressed  with  some  kind  of  protec¬ 
tion.  The  committee  believes  that  the  consideration  of  ord¬ 
nance  in  the  assessment  of  vulnerability,  both  in  modeling 
and  in  testing,  is  essential.3 

It  is  the  committee’s  opinion  that  the  policy  level  of  the 
Army  supports  limited  full-scale,  full-up  testing,  but  that 
this  support  has  not  been  fully  recognized  at  the  program 
level. 

The  Navy  LFT&E  Programs 

The  Navy  Live  Fire  Test  philosophy  and  the  program  for 
the  V-22  were  presented  to  the  committee  by  a  representative 
of  the  Survivability  Branch,  Naval  Air  Systems  Command. 


This  aspect  of  vulnerability  is  examined  in  Chapter  2. 
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The  Deputy  Director  of  the  Navy’s  Office  of  Test  &  Evalua¬ 
tion  and  Technology  Requirements  was  present  at  the 
briefing.  Preliminary  information  on  the  LFT&E  program 
for  the  A- 12  aircraft  was  provided  during  several  committee 
meetings.  The  Navy’s  presentation  is  summarized  below. 

The  Navy  Live  Fire  Test  and  Evaluation  Policy.  The 
Navy’s  view  of  the  Live  Fire  Test  law  is  that  it  encourages 
full-up  vulnerability  testing,  under  realistic  combat  condi¬ 
tions,  first  at  the  sub-scale  level  and  later  at  the  full-scale 
level.  Live  fire  testing  is  part  of  developmental  testing  and 
is  a  continuing  process  that  contributes  to  the  development 
and  engineering  of  Navy  aircraft  The  Navy  policy  is  to 
comply  with  the  intent  of  the  law  by  establishing  realistic 
operational  requirements  and  combat  threat  scenarios,  estab¬ 
lishing  explicit  survivability  and  vulnerability  design  re¬ 
quirements,  and  requiring  Live  Fire  Testing  and  Evaluation 
as  part  of  the  development  process  using  a  shoot-fix-shoot- 
again  approach.  The  Navy  LFT&E  program  integrates  Live 
Fire  Testing  with  the  development  of  the  aircraft,  as  shown 
in  Figure  4-2.  The  integration  of  the  tests  on  simulators  and 
sub-scale  hardware  early  in  the  development  provides  design 
information  in  time  to  be  useful.  Later  Live  Fire  Verification 


Testing  using  simulators  and  surrogates  provides  informa¬ 
tion  on  the  design’s  compliance  with  the  specifications. 

According  to  the  Navy  presenter,  the  benefits  identified 
for  full-scale,  fiill-up  testing  are  the  possible  discovery  of 
previously  unknown  reactions  to  the  weapons  and  synergis¬ 
tic  effects.  The  disadvantages  include  the  fact  that  the  results 
are  very  limited  statistically;  the  full-scale  aircraft  is  at  risk 
on  each  test;  the  tests  are  very  costly;  and  the  full-scale, 
full-up  tests  do  not  eliminate  the  need  for  all  of  the  earlier 
sub-scale  tests.  Furthermore,  testing  the  full-scale  aircraft 
can  be  accomplished  only  at  a  very  late  stage  in  the  develop¬ 
ment,  and  it  may  not  be  possible  or  practical  to  correct  any 
design  vulnerabilities  discovered  in  the  tests.  This  is  contrary 
to  the  requirement  in  the  law  that  the  testing  shall  be  carried 
out  sufficiently  early  to  allow  any  design  deficiency  demon¬ 
strated  by  the  testing  to  be  corrected  before  proceeding  be¬ 
yond  low-rate  initial  production.  In  summary.  Live  Fire  Test¬ 
ing  is  a  process  that  begins  in  the  D/V  phase;  it  is  not  a 
single  pass/fail  event  It  involves  both  analyses  and  testing, 
with  results  available  sufficiently  early  to  aid  in  design. 

Live  Fire  Testing  of  the  V-22  Aircraft  A  comprehensive 
plan  for  the  Live  Fire  Testing  of  the  V-22  was  made  available 
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to  the  committee,  and  though  it  is  somewhat  dated  it  was  a 
useful  input.  The  program  is  old  enough  to  be  grandfathered 
for  Live  Fire  Testing  even  though  the  summary  page  says 
the  current  program  meets  the  Live  Fire  Test  law. 

The  V-22  aircraft  had  specific  design  requirements  on 
the  allowable  vulnerable  area  for  several  specific  threat 
weapons,  excluding  hydraulic  ram.  A  number  of  alternative 
vulnerability  reduction  designs  were  identified,  and  the  re¬ 
duction  in  vulnerable  area  and  increase  in  weight  were  deter¬ 
mined  for  each  alternative.  The  most  promising  alternatives 
were  tested.  Some  of  the  test  articles  used  in  the  stage  I 
ballistic  tests  included  a  wing  structure,  a  fuselage  structure, 
a  pylon  actuator,  and  some  of  the  flight  controls.  Ballistic 
verification  tests  have  been  completed  for  phase  I  and  phase 
II  wing  tank  hydraulic  ram  and  the  propellar  rotor  gearbox 
components.  A  series  of  dry  bay  fire  suppression  tests  has 
been  conducted  using  a  representative  three-bay  simulator. 
Four  different  suppression  techniques  were  tested;  a  Halon 
system,  power  panels,  foam,  and  an  on-board  inert  gas  gener¬ 
ating  system  (OBIGGS).  The  threats  were  several  ballistic 
projectiles.  In  the  baseline  tests  without  any  protection,  large, 
consistent  fires  with  heavy  fixture  damage  were  observed. 
The  results  using  the  different  suppression  techniques  ranged 
from  several  fires  to  no  fires. 


Committee  Comments  on  the  V-22 ,  The  V-22  aircraft  has 
been  under  development  for  several  years  and  has  been 
under  a  cloud  for  many  of  them.  The  cloud  is  related  to 
the  need  for  the.  system  rather  than  anything  related  to  its 
vulnerability.  The  cancellation  of  the  program  by  the  Secre¬ 
tary  of  Defense  and  the  reinstatement  of  the  program  by 
Congress  tend  to  make  systematic  planning  for  various 
phases  of  Live  Fire  Testing  difficult 

The  purpose  of  the  aircraft  is  to  take  troops  and  light 
material  into  the  vicinity  of  danger,  and  for  that  mission  the 
Joint  Services  Operational  Requirement  for  ballistic  toler¬ 
ance  seems  inadequate.  Furthermore,  there  is  no  discussion 
of  the  vulnerable  area  associated  with  on-board  ordnance. 
The  aircraft  will  be  hauling  U.S.  Marines  into  combat,  and 
they  will  put  all  of  the  weapons  and  ordnance  they  can 
load  aboard.  An  analysis  that  shows  how  the  load  could  be 
distributed,  or  how  some  light  armoring  could  change  the 
vulnerability  of  the  aircraft  to  ground  fire,  would  give  more 
credence  to  the  claim  that  the  ballistic  protection  had  been 
achieved 

The  fate  of  the  LFT&E  program  is  dependent  upon  a 
decision  to  enter  into  low-rate  production. 

•  i 

Live  Fire  Testing  of  the  A-12  Aircraft  The  A-12  was  a 
highly  classified  program,  and  therefore,  it  was  not  a  covered 
system  according  to  the  LFT  law.  Nevertheless,  a  member 


of  the  LFT  Office  was  read  into  the  program  because  of  the 
fact  that  the  aircraft  would  become  a  candidate  for  LFT 
when  it  no  longer  was  a  black  program.  According  to  one 
of  the  briefers,  the  A-12  live  fire  test  program  for  components 
and  subsystems  was  one  of  the  best  programs  to  date.  Al¬ 
though  the  proposed  LFT  strategy  did  not  include  testing 
on  the  full-scale,  full-up  aircraft,  it  appeared  to  be  reasonable 
according  to  the  briefer.  Apparently,  informal  approval  was 
given  to  the  proposed  LFT  strategy,  and  this  strategy  was 
in  effect  when  the  program  was  canceled.  Two  major  criti¬ 
cisms  of  the  program  were  the  omission  of  testing  of  the 
on-board  ordnance  and  the  omission  of  consideration  of 
threats  other  than  projectiles  and  missiles,  such  as  directed 
energy  weapons. 

Committee  Comments  on  the  A-12 .  It  is  the  committee’s 
opinion  that  the  policy  levels  of  the  Navy  and  the  Program 
Offices  appear  to  have  no  intention  of  doing  any  full-scale, 
full-up  test  on  any  aircraft  before  proceeding  beyond  low- 
rate  initial  production.  The  Navy  considers  full-scale,  full- 
up  testing  of  aircraft  to  be  unreasonably  expensive  and  im¬ 
practical.  Apparently  as  a  consequence  of  this  belief  and  the 
interpretation  of  the  LFT  requirements  given  in  the  LFT&E 
Guidelines,  no  full-scale,  full-up  LFT  tests  were  proposed 
for  the  A-12.  The  committee’s  opinion  is  that  any  approval 
of  an  LFT  strategy  for  the  A-12  that  did  not  include  tests 
on  the  full-scale,  full-up  aircraft  and  did  not  request  a  waiver 
from  these  tests  would  have  been  in  error.  The  committee 
is  concerned  that  future  black  programs  will  face  the  same 
problems  with  respect  to  the  LFT  law  that  occurred  with 
the  A-12.  The  Navy  has  initiated  efforts  on  a  replacement 
program  for  the  A-12,  presently  called  the  AX.  Because  this 
also  is  a  black  program,  it  is  not  a  covered  system  under 
the  LFT  law.  However,  it  too  will  eventually  become  an 
unclassified  program  and  at  that  time  will  be  subject  to  the 
law’s  provisions.  This  will  probably  occur  after  the  program 
enters  into  full-scale  engineering  development,  the  deadline 
for  the  application  for  a  waiver.  If  the  requirements  in  the 
LFT  law  are  ignored  until  the  program  comes  out  of  the 
black,  there  most  likely  will  be  a  confrontation  with  the  LFT 
Office. 

The  Air  Force  LFT&E  Programs 

The  Air  Force  Live  Fire  Test  Policy .  The  Air  Force  po  licy 
on  LFT&E  was  presented  by  a  representative  from  Test  & 
Evaluation,  Air  Force,  and  was  entitled  “Air  Force  Policy 
Considerations  for  Live  Fire  Test  &  Evaluation.”  According 
to  the  presenter,  the  objective  of  Air  Force  LFT&E  is  to 
provide  a  timely  and  thorough  assessment  of  the  vulnerabil¬ 
ity  and  lethality  of  a  system  as  it  progresses  through  its 
development.  This  is  accomplished  by  a  balanced  program 
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of  analysis  and  test.  It  is  a  systems  approach,  similar  to 
development  testing  used  in  other  areas.  Because  it  is  prohib¬ 
itive  to  test  ail  possible  combinations  of  threat/aircraft/condi¬ 
tions,  analysis  must  be  an  integral  part  of  the  LFT&E  proc¬ 
ess.  A  general  sequence  of  design-analyze-test  is  followed 
and  repeated  if  necessary.  The  analytical  models  are  those 
accepted  by  the  Joint  Technical  Coordinating  Group  on  Air¬ 
craft  Survivability  (JTCG/AS).  The  LFT&E  program  is  initi¬ 
ated  sufficiently  early  to  allow  the  results  to  impact  the 
system  design.  The  benefits  of  LFT&E  can  be  maximized 
only  when  the  test  results  are  used  to  aid  in  the  design.  The 
LFT&E  strategy,  which  is  included  in  the  TEMP,  ensures 
that  all  issues  are  addressed  and  integrated  with  the  other 
elements  of  the  system  program.  According  to  the  presenter, 
the  Air  Force  policy  is  in  compliance  with  the  reporting 
requirements  of  the  LFT&E  Office.  The  threat  used  for  the 
Live  Fire  Tests  is  the  threat(s)  defined  by  the  Operational 
Requirements  Document  (ORD).  The  ORD  is  the  official 
source  of  user  requirements,  and  the  ORD  process  places  the 
user  in  the  role  of  integrating  the  System  Threat  Assessment 
Report  (STAR),  aircraft  characteristics,  employment  con¬ 
cepts,  etc.,  to  derive  the  expected  threat. 

According  to  the  presenter,  the  test  hardware  will  be 
of  sufficient  size  and  quantity  so  that  realistic  test  results, 
including  synergism  and  secondary  effects,  will  be  obtained. 
Experience  indicates  that  components,  subsystems,  and  sub- 
assemblies  are  generally  sufficient  and  more  desirable  than 
a  complete  system.  Full  consideration  will  be  given  to  the 
use  of  actual  hardware,  replicas,  and  surrogates.  However, 
use  of  these  items  must  depend  on  the  technical  payoff, 
availability,  and  cost.  The  requirements  for  additional  live 
fire  test  facilities  are  being  identified  in  the  Air  Force’s  Test 
Investment  Planning  Process.  In  summary,  according  to  the 
Air  Force,  its  LFT&E  program  is  integral  to  the  system 
design,  development,  test,  and  evaluation  process.  The  Air 
Force  believes  it  endorses  an  intelligent  approach  to  LFT&E 
that  considers  all  variables  affecting  aircraft  effectiveness. 
It  believes  the  Air  Force  T&E  approach  employs  the  most 
prudent  combination  of  analyses  and  tests,  is  viable,  and  is 
producing  positive  results. 


The  Live  Fire  Test  Program  for  the  F-22  Aircraft  The 
Live  Fire  Test  program  was  briefed  to  the  committee  by  a 
representative  from  the  system  program  office  in  a  presenta¬ 
tion  entitled  “F-22  Vulnerability  Program.”  The  approach 
to  aircraft  survivability  used  by  the  F-22  is  a  combination 
of  low  susceptibility  and  reduced  vulnerability.  Directed 
energy  weapons  are  considered,  as  well  as  the  conventional 
gun  and  missile  threats.  The  reduction  in  vulnerability  is 
accomplished  through  incorporation  of  redundant  subsys¬ 
tems  and  damage  tolerant  features.  Vulnerability  analyses 
will  be  used  to  establish  any  system  weaknesses  and  to 


identify  vulnerability  reduction  candidates.  Vulnerability 
testing  of  materials,  components,  and  subassemblies  is  used 
to  verify  the  vulnerability  reduction  candidates  and  to  iden¬ 
tify  issues  that  necessitate  Live  Fire  Testing.  The  results 
from  the  analyses  and  the  lower-level  tests  will  determine 
the  need  for,  and  extent  of,  further  validation  tests.  According 
to  the  presenter,  the  LFT&E  program  is  a  sound  development 
and  demonstration  program  that  represents  a  balance  of  the 
technical  merits,  cost,  and  schedule.  The  methodology  used 
in  the  program  is  shown  in  Figure  4-3,  and  the  vulnerability 
program  schedule  and  funding  are  given  in  Figure  4-4.  Ac¬ 
cording  to  the  presenter,  the  budget  and  schedule  supports 
the  conduct  of  prudent  Live  Fire  Testing.  Some  of  the 
assessments  to  be  conducted  to  verify  the  design  include 
fuel  tank  inerting  using  a  rig  test,  redundancy  and  separation 
studies  using  analyses  and  inspection,  verification  of  the 
fragmentation  resistance  of  pressure  vessels  using  ballistic 
tests,  and  a  static  test  of  structure. 

Committee  Comments  on  the  F-22.  In  the  F-22  presenter’s 
discussion  of  vulnerability  modeling  for  the  engineering, 
manufacturing  and  development  (EMD)  study  he  stated  that 
internal  armament  was  treated.  However,  a  chart  of  vulnera¬ 
bility  requirements  included  a  column  for  armaments  that 
discussed  vulnerable  area  as  to  be  determined  (TBD)  and 
fire  inerting,  fire  protection,  and  shielding  as  not  applicable 
(N/A).  The  EMD  vulnerability  program  shown  in  Figure 
4-5  has  a  category  for  stored  munitions,  engine,  and  aircraft 
availability,  but  it  is  under  concurrent  JLF.  When  members 
of  the  committee  questioned  the  possibility  that  this  was  the 
prototype  aircraft,  the  briefer  seemed  unaware  that  a  Live 
Fire  Test  would  be  done  on  a  full-scale,  full-up  aircraft,  and 
that  the  prototypes  were  so  different  from  the  production 
model  that  they  would  not  be  representative. 

The  Live  Fire  Test  Program  for  the  C-17  Aircraft  The 
program  for  the  C-17  LFT&E  was  given  by  a  representative 
from  the  C-17  System  Program  Office  in  a  presentation 
entitled  “C-17,  Live  Fire  Test  (LFT)  Program.”  According 
to  the  presenter,  the  C-17  is  expected  to  be  deployed  to 
within  20  to  40  km  of  the  Forward  Edge  of  the  Battle 
Area  (FEB  A).  The  vulnerability  requirement  for  the  selected 
design  threat  in  the  FEBA  is  met  by  using  redundancy 
with  separation,  damage  tolerant  components,  and  fire  and 
explosion  suppression.  A  number  of  design  changes  were 
made  to  meet  the  requirement  as  the  result  of  analytical 
assessments.  These  included  the  relocation  of  the  oxygen 
converter,  rerouting  of  the  hydraulic  lines,  and  changes  to 
the  pitch  trim  actuator  and  aileron  hinge  fitting.  The  test 
methodology  included  design,  analysis,  test,  redesign,  and 
retest  The  test  articles  included  components,  such  as  crew 
armor,  pressure  vessels,  the  upper  wing  skin,  and  a  flap 
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FIGURE  4-3  Vulnerability  assessment  methodology  for  the  F-22. 
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FIGURE  4-4  Vulnerability  program  for  the  F-22. 


hinge  fitting;  and  a  functional  8-foot  section  of  the  wing 
leading  edge.  The  leading  edge  section  is  to  be  Live  Fire 
Tested  to  determine  if  there  is  a  leading  edge  fire  problem. 
The  test  article  is  designed  from  production  drawings  from 
the  Douglas  Aircraft  Company  and  will  be  built  by  the 
4950th  Test  Wing  Modification  Branch.  The  Live  Fire 
Tests  will  be  conducted  by  the  Wright  Laboratory's  Flight 
Dynamics  Directorate. 


Committee  Comments  on  the  C*17.  The  C-17  represents 
one  of  the  most  interesting  challenges  to  the  Live  Fire  Test 
law  of  1987.  It  was  in  full  scale  development  when  the  law 
was  passed  and  hence  could  not  legally  request  a  waiver.  It 
is  a  very  large  and  expensive  aircraft,  although  not  the  largest 
to  enter  the  inventory.  The  largest,  the  C-5 A,  was  delivered 
during  the  1960s  and  was  not  subject  to  any  live  fire  testing. 
Apparently,  Congress  considers  the  C-17  to  be  a  Live  Fire 


Test  candidate  and  expects  to  see  Live  Fire  Testing  con¬ 
ducted  on  the  C-17  aircraft  (Bennett,  1991).  The  contention 
by  Congress  is  that  testing  of  components  from  a  military 
vehicle  in  an  essentially  inert  condition  is  no  substitute  for 
the  firing  of  threat  ammunition  at  that  vehicle  loaded  for 
combat  with  the  intended  fuels,  fluids,  and  ordnance  on 
board  and  in  place. 

In  the  case  of  cargo  aircraft  such  as  the  C-17,  if  the 
aircraft  contains  ordnance  when  configured  for  combat,  the 
ordnance  and  its  reaction  to  ballistic  impact  must  be  treated 
in  the  assessment  A  sub-scale,  but  full-up.  Live  Fire  Test 
of  the  C-17  would  logically  consist  of  testing  a  mock-up  of 
an  actual  cargo  compartment  with  ammunition  loaded  in 
pallets  on  dunnage.  If  the  shot  created  significant  damage 
due  to  a  reaction  of  the  ammunition,  shielding  or  armoring 
could  be  undertaken  to  bring  the  damage  expected  to  an 
acceptable  level. 

During  a  temporary  shifting  around  of  personnel  in  the 
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UVE  FIRE  TEST  PROGRAMS 

Office  of  the  Deputy  Director,  Defense  Research  and  Engi¬ 
neering  (Test  &  Evaluation),  the  Air  Force’s  proposal  to  test 
only  a  portion  of  the  leading  edge  of  the  wing  was  accepted 
as  being  in  compliance  with  the  LFT  law.  This  enforced  the 
belief  that  the  conduct  of  full-scale,  full-up  Live  Fire  Testing 
could  be  avoided  by  the  acquisition  arm  of  the  Air  Force. 
The  Air  Force  user  should  have  been  put  more  of  a  decision¬ 
making  position  on  this  issue.  The  size  of  the  C-17  is  going 
to  make  it  an  attractive  target  for  various  potential  enemies 
who  might  hold  fire  from  smaller  targets  in  order  to  concen¬ 
trate  fire  on  the  larger  aircraft.  If  an  aircraft  is  to  go  into 
combat,  and  if  combat  damage  is  expected  to  be  encountered, 
the  aircraft  is  going  to  have  to  be  tolerant  of  the  damage 
or  a  price  will  be  paid  in  manpower  and  material  losses. 
Supporters  of  the  Live  Fire  Test  law  believe  that  if  the  full- 
scale  C-17  is  too  expensive  to  be  subjected  to  live  fire  in  a 
full-up  condition,  the  aircraft  is  too  expensive  to  risk  losing 
by  delivering  cargo  to  a  forward  military  zone.  If  that  is  the 
case,  the  upper  limit  on  the  cost  a  Service  may  be  willing 
to  put  into  an  aircraft  may  have  been  reached.  A  reasonable 
question  to  ask  is,  would  the  aircraft  be  bought  if  it  was  to 
be  used  only  to  deliver  military  cargo  to  rear  areas?  If  that 
answer  is  no,  the  importance  of  testing  the  aircraft  against 
the  weapons  it  is  expected  to  encounter  is  paramount 

The  Test  Community’s  View  of  Live  Fire  Testing 

The  live  fire  test  community  of  the  three  Services  appears 
to  be  in  general  agreement  about  the  efficacy  of  the  Live 
Fire  Test  law  as  it  applies  to  aircraft.  Members  of  that 
community  do  not  consider  it  necessary  to  do  full-scale, 
full-up  tests  in  order  to  determine  most  of  the  design  vulnera¬ 
bilities.  However,  they  do  consider  it  essential  to  conduct 
many  sub-scale  tests  on  components  and  sub-systems,  both 
inert  and  full-up,  in  the  development  cycle  of  an  aircraft 
The  sub-scale  test  results  are  invaluable  sources  of  informa¬ 
tion  that  can  be  used  initially  to  aid  the  design  and  later  to 
verify  the  design.  The  test  community  believes  that  the  full- 
scale,  full-up  tests  are  conducted  too  late  in  the  development 
cycle  to  be  of  much  value  to  the  designer  and  that  the  amount 
of  information  obtained  from  the  tests  is  very  limited.4  Every 
time  a  test  is  conducted  a  portion  of  the  test  article  is  dam¬ 
aged,  and  it  may  be  impossible  to  restore  the  article  to  its 
original  condition.  Furthermore,  these  tests  require  very 
large  budgets  that  could  go  toward  the  expansion  of  test 
facilities  and  capabilities  and  the  conduct  of  many  more 
sub-scale  tests.  The  test  community  does  recognize  the  possi¬ 
bility  of  an  unanticipated  reaction,  cascading  damage,  or 
synergism  occurring  in  the  full-scale  aircraft  However,  its 


*The  test  community  is  not  against  all  testing  of  full-scale,  full*  up 
aircraft;  it  fully  supports  this  type  of  testing  on  existing  aircraft  in  programs 
such  as  JLF. 
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members  believe  that  nearly  all  of  the  kill  modes  of  an 
aircraft  are  known  and  can  be  anticipated;  they  state  that 
they  very  seldom  see  a  test  result  that  was  unanticipated. 
The  magnitude  of  the  response  to  the  hit  may  be  larger  or 
smaller  than  expected,  but  the  response  was  anticipated.5 

Committee  Comments .  The  committee  notes  that  not 
everyone  who  has  observed  live  fire  tests  on  sub-scale  and 
full-scale  test  articles  shares  the  views  held  by  these  testers. 
They  believe  that  there  have  been  unanticipated  results  from 
these  tests.  Furthermore,  even  when  the  response  is  as  ex¬ 
pected,  the  difference  in  the  expected  magnitude  of  the  re¬ 
sponse  and  the  observed  magnitude  often  is  too  large  to  be 
acceptable.  The  testers  have  also  overlooked  the  fact  that 
information  from  the  full-scale  tests  is  an  input  to  acquisition 
decision  makers  at  milestone  reviews.  The  committee  also 
believes  that  the  test  community  has  not  given  proper  consid¬ 
eration  to  the  on-board  ordnance  problem. 

The  Industry  View  of  Live  Fire  Testing 

The  U.S.  aircraft  industry  does  not  play  a  major  role 
in  LFT&E.  However,  the  committee  solicited  its  opinion 
because  these  are  the  users  of  the  results  of  the  Live  Fire 
Tests.  The  four  industry  representatives  were  unanimous  in 
their  belief  that  the  LFT  law  as  described  in  the  LFT  Guide¬ 
lines  and  Planning  Guide  does  not  require  a  full-scale,  full- 
up  aircraft  to  be  tested.  Furthermore,  they  do  not  believe  it 
should.  One  of  the  major  reasons  they  gave  for  not  recom¬ 
mending  full-scale  testing  of  a  “production”  aircraft  is  that 
the  design  is  usually  frozen  by  the  time  the  results  become 
available.6  However,  they  were  in  full  agreement  with  the 
test  community  on  the  necessity  of  sub-scale  testing  through¬ 
out  the  program  development. 

Controversy  Regarding  Which  Munitions  to  Use  in  the 
Live  Fire  Testing  Program 

The  Live  Fire  Test  law  stipulates  that  “ munitions  likely 
to  be  encountered  in  combat  (or  munitions  with  a  capability 
similar  to  such  munitions?'  are  to  be  used  in  the  “ realistic 
survivability  testing  ”  The  specific  munitions  to  be  used  in 
the  Live  Fire  Testing  of  a  particular  aircraft  are  selected  by 
the  aircraft  Program  Office  as  part  of  its  LFT&E  program 
plan.  Typically,  the  threats  selected  by  the  Program  Office 
for  Live  Fire  Testing  are  the  threats  the  aircraft  was  designed 
to  withstand,  such  as  a  single  hit  by  an  armor-piercing  (API) 


5This  attitude  appears  to  be  in  contrast  to  the  experience  of  the  testers 
of  ground  vehicles,  where  the  particular  response  to  a  hit  was  often  unantici¬ 
pated. 

*The  committee  notes  that  industry  shares  the  same  reluctance  as  the 
Services  to  change  a  design  late  in  the  development  cycle. 
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or  smail-caliber  high  explosive  with  incendiaries  (HEI).  The 
assumption  is  made  that  the  more  lethal  overmatching 
threats,  such  as  the  larger  anti-aircraft  artillery  (AAA)  and 
the  surface-to-air  missiles  (SAMs),  will  be  avoided  and 
hence  should  not  be  a  part  of  the  LFT&E  program.  For 
example,  the  Navy  and  the  Air  Force  are  currently  devel¬ 
oping  stealth  aircraft  whose  survivability  is  strongly  related 
to  not  getting  hit.  They  believe  that  some  “credit”  should 
be  given  for  this  aspect  of  survivability.  Aircraft  that  do  not 
get  hit  as  often  do  not  have  to  rely  as  much  on  their  reduced 
vulnerability  to  survive.  Furthermore,  if  these  overmatching 
threats  are  included  in  the  LFT&E  program,  they  could 
destroy  the  only  test  article(s)  available,  as  well  as  the  test 
equipment,  and  perhaps  some  of  the  test  facilities. 

The  Live  Fire  Test  Office  has  interpreted  the  phrase  “mu¬ 
nitions  likely  to  be  encountered  in  combat”  to  mean  that 
those  munitions  the  aircraft  may  encounter,  including  the 
latest  directed  energy  weapons,  should  be  included  in  the 
LFT&E  program,  regardless,  of  the  design  threat  for  the 
aircraft.  This  interpretation  of  the  phrase  “likely  to  be  en¬ 
countered  in  combat”  made  by  the  LFT  Office  conflicts  with 
the  interpretation  made  by  the  Program  Managers,  and  this 
difference  in  interpretations  has  been  the  source  of  consider¬ 
able  controversy.  The  LFT  Office  argues  that  the  user  Ser¬ 
vice’s  intention  to  avoid  the  most  lethal  threats  in  combat 
through  susceptibility  reduction  may  not  always  be  achiev¬ 
able.  Given  that  the  intent  of  the  LFT  law  is  not  to  intention¬ 
ally  destroy  aircraft,  the  LFT  Office  believes  that  a  test 
involving  a  “potentially”  overmatching  weapon  can  be  de¬ 
signed  that  will*  provide  information  on  any  design  weak¬ 
nesses  with  little  likelihood  of  destroying  the  aircraft.  For 
example,  suppose  the  weapon  is  a  guided  missile  with  a 
large  HE  warhead  and  a  proximity  fuze.  The  lethality  of 
this  weapon  depends  upon  the  location  of  the  warhead  with 
respect  to  the  aircraft  at  the  time  of  detonation.  If  the  detona¬ 
tion  occurs  next  to  the  aircraft,  the  aircraft  will  most  likely 
be  killed.  On  the  other  hand,  if  the  detonation  occurs  far 
from  the  aircraft,  the  aircraft  will  most  likely  not  be  killed 
Consequently,  the  LFT  Office  believes  that  if  information 
on  any  design  weaknesses  can  be  obtained  from  a  distant 
detonation  with  little  probability  of  destroying  the  aircraft, 
the  test  should  be  considered  The  Services  counter  this 
argument  with  the  argument  that  any  information  obtained 
from  the  limited  number  of  tests  is  not  statistically  meaning¬ 
ful.  This  is,  in  turn,  countered  by  the  argument  that  one 
purpose  of  the  full-scale  tests  is  to  discover  vulnerabilities, 
not  to  quantify  vulnerability. 

Committee  Comments .  Today,  military  systems  are  de¬ 
signed  to  survive  the  threat  they  can  expect  to  find  in  the 
field  at  the  time  they  are  fielded7  The  process  of  defining 


’Aircraft  have  not  always  had  specific  design  requirements  to  survive 


the  future  threat  is  called  threat  projection.  The  results  of 
the  threat  projection  appear  in  STAR.  Historically,  the  threat 
community  has  been  reluctant  to  commit  to  a  firm  projection 
of  the  specific  threats  a  particular  system  will  encounter  10 
to  20  years  in  the  future  for  several  reasons.  Consequently, 
a  long  list  of  threats  that  the  system  may  encounter  in  combat 
is  usually  prepared,  perhaps  with  some  prioritization.  How¬ 
ever  flawed  the  threat  projection  process  may  be,  the  user 
of  the  proposed  system  must  select  a  design  threat  for  the 
system.  The  developer  then  designs  a  system  that  will  survive 
the  threat  described  in  the  user’s  requirements  document.8 

Note  that  the  user  is  the  organization  that  examines  the 
threat  projection  and  selects  the  design  threat.  The  design 
threat  selected  may  or  may  not  be  the  most  likely  threat 
that  will  be  encountered;  usually  it  is  but  one  of  many 
threats  that  are  likely  to  be  encountered.  The  Live  Fire 
Test  Office  does  not  consider  the  design  threat  selected  by 
the  user  as  the  only  threat  likely  to  be  encountered  and 
strongly  encourages  the  Program  Office  to  test  the  system 
against  the  other  threats  that  exist,  or  will  exist,  in  the 
operating  environment.  The  argument  given  by  the  LFT 
Office  is  that  the  threat  selected  for  design  is  usually  one 
that  can  be  defeated  without  significant  increases  in  weight 
and  cost.  Thus,  given  that  it  is  important  to  verify  the 
design’s  ability  to  withstand  the  design  threat,  it  is  also 
important  to  determine,  and  correct  if  possible,  any  weak¬ 
nesses  in  the  design  when  subjected  to  the  other  “likely 
to  be  encountered”  threats  before  the  system  goes  beyond 
LRIP.  An  argument  against  testing  against  the  other,  possi¬ 
bly  overmatching,  threats  is  that  even  though  the  system 
was  not  required  to  defeat  these  weapons,  any  negative 
results  from  such  tests  may  jeopardize  the  program,  as  well 
as  siphon  off  much  needed  funding. 

The  C-17  LFT&E  program  is  an  example  of  the  conflict 
between  testing  only  against  the  design  threat  and  testing 
against  other  threats  that  may  be  encountered  in  combat. 
The  Operational  Requirements  Document  for  the  C- 1 7  stipu¬ 
lates  a  design  that  will  withstand  a  single  hit  by  a  certain 
ballistic  projectile.  According  to  the  C-17  presenter,  this 
projectile  may  be  encountered  by  the  C-17  in  the  region  of 
20  to  40  km  from  the  Forward  Edge  of  the  Battle  Area.  The 
Live  Fire  Test  Office,  on  the  other  hand,  believes  that  C- 
17s  in  the  forward  area  of  the  battlefield  and  at  austere 
landing  sites  near  combat  zones  can  expect  threats  that  are 


the  expected  threat  particularly  on  the  vulnerability  aspect  of  survivability. 
Perhaps  the  first  aircraft  to  have  a  vulnerability  requirement  on  the  design 
were  the  U.S.  Army’s  UTTAS  and  AAH.  now  the  UH-fiO  Black  Hawk 
and  AH-64  Apache.  Both  aircraft  had  to  be  designed  to  survive  a  single 
hit  by  a  non-explosive  ballistic  projectile  anywhere  on  the  aircraft  and  fly 
for  30  minutes  after  the  hit 

‘As  a  consequence  of  this  procedure,  systems  are  typically  designed 
against  today’s  threats — and  sometimes  even  against  yesterday’s  threats. 
Both  practices  result  in  systems  being  designed  to  •survive  yesterday’s 
threats  when  they  finally  appear  in  the  field. 
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more  much  lethal  than  the  design  threat  selected  for  the  C- 
17.’ 


A  Program  Manager’s  View  of  the  Live  Fire  Test  Law 

The  committee,  after  listening  to  all  of  the  presentations, 
believes  the  following  description  of  a  Program  Manager’s 
(PM’s)  view  of  the  LFT  law  may  be  representative  of  current 
opinion: 

The  full-scale,  full- up  testing  mandated  by  the  LFT  law 
is  an  unquantifiable,  but  potentially  catastrophic,  risk  to  his 
program.  LFT  has  no  quantitative  contractual  specification 
or  acceptance  criteria  at  program  initiation.  No  quantitative 
criteria  for  acceptable  or  unacceptable  damage  are  included 
in  the  requirements  process,  milestone  commitments,  or  con¬ 
tractor  performance  documents.  LFT  of  the  full-scale  aircraft 
occurs  late  in  the  development  phase  of  the  program,  and 
there  may  be  neither  adequate  time  nor  money  to  conduct 
the  tests  or  to  make  any  changes  required  as  a  result  of  the 
tests.  Further,  the  PM  may  believe  that  neither  the  definition 
of  the  tests,  nor  the  conduct  of  the  tests,  nor  the  interpretation 
of  the  test  results  is  totally  under  his  control.  The  perceived 
jeopardy  to  his  program  created  by  LFT  is  exacerbated  by 
the  severe  requirement  to  fit  the  program  into  a  somewhat 
inflexible  overall  resource  schedule,  both  in  time  and  in 
dollars.  In  summary,  LFT  represents  a  considerable  source 
of  problems  to  the  PM,  in  the  form  of  an  uncontrollable, 
potentially  catastrophic  uncertainty,  as  he  attempts  to  suc¬ 
cessfully  complete  the  development  of  his  system,  and 
should  be  avoided  if  at  all  possible. 

Issues  Relating  to  Distrust  Among  the  Participants  in 
Live  Fire  Testing 

The  committee  is  aware  of  the  strong  differences  of  opin¬ 
ion  held  by  various  individuals  and  organizations  concerning 
the  efficacy  of  the  Live  Fire  Test  law  and  of  the  level  of 
mutual  distrust  that  has  evolved  as  a  result  of  these  opinions. 
This  distrust  between  the  various  participants  of  each  other’s 
motives  and  actions  is  probably  responsible  for  the  ever- 
increasing  tensions  within  the  current  Live  Fire  Test  pro¬ 
gram.  The  attitudes  of  the  major  participants  concerning  the 
Live  Fire  Test  law  and  its  place  in  the  acquisition  process 
appear  to  be  those  described  below. 

In  the  committee’s  opinion,  the  Services  may  believe 
that,  as  system  developers  and  users,  they  know  what  is 
needed  in  the  equipment  they  will  take  into  the  field,  and 


^Military  airlift crs  have  flown  in  hostile  environments  in  the  past  Exami¬ 
nation  of  the  combat  data  from  the  Southeast  Asia  conflict  reveals  damage 
and  losses  to  C-7$,  C-123s,  and  C-130s  from  small  arms.  AAA.  and  shoul¬ 
der-fired  SAMs,  and  14  airlifters  were  damaged  by  small  arms  in  only  400 
sorties  in  1989  in  Operation  Just  Cause  (Ropelewski.  1990). 


that  they,  the  Services,  are  directly  responsible  for  the  fate 
of  the  military  personnel  who  use  this  equipment  in  combat. 
They  are  very  apprehensive  about  any  outside  organization 
that  can  dilute  their  ability  to  define  the  necessary  equipment 
testing  and  the  procedures  required  to  accomplish  this  test¬ 
ing.  They  appear  to  further  believe  that  the  Live  Fire  Test 
law  gives  to  others  not  directly  responsible  for  the  delivered 
product  inordinate  control  without  any  accompanying  re¬ 
sponsibility  for  the  quality  of  the  product  or  its  cost. 

In  the  committee’s  opinion,  the  Office  of  the  Secretary 
of  Defense  (OSD)  may  believe  that  there  have  been  a  suffi¬ 
ciently  large  number  of  prior  experiences  in  the  area  of  live 
fire  testing  to  indicate  that  pressure  by  a  Service  for  success¬ 
ful  and  rapid  certification  of  its  products  under  development 
can  lead  to  inadequate  live  fire  testing  and  to  subsequent 
unnecessary  combat  vulnerabilities.  The  OSD,  therefore,  has 
chosen  to  exercise  close  control  over  the  Live  Fire  Test 
programs  and  assumes  the  ultimate  authority  for  approval 
of  the  equipment  based  on  the  program  results. 

In  the  committee’s  opinion,  the  Congress  may  believe 
that  there  are  sufficient  numbers  of  proven  instances  of  un¬ 
necessary  combat  vulnerability  in  the  Department  of  Defense 
(DoD)  equipment  previously  delivered  to  the  field  to  warrant 
legislative  direction  of  DoD  test  certification  to  include  live 
fire  testing  of  full-scale,  full-up  systems  using  munitions 
likely  to  be  encountered  in  combat.  Congress  further  believes 
that  it  has  the  ultimate  responsibility  for  the  programs  it 
authorizes  and  therefore  has  the  obligation  to  exercise  that 
legislative  direction. 

Conclusion 

The  committee  believes  the  common  view  of  all  parties 
knowledgeable  in  this  business  is  that  the  current  assessment 
procedure,  including  both  analysis  and  Live  Fire  Testing, 
does  not  guarantee  that  the  U.S.  armed  forces  will  field  cost- 
effective  systems  designed  for  reduced  vulnerability.  The 
intent  of  the  LFT  law  to  contribute  to  the  creation  of  less 
vulnerable  aircraft  designs  is  valid;  its  execution  to  achieve 
this  intent  has  been  flawed  in  several  ways  as  identified  in 
Chapters  2,  3,  and  4.  The  committee  believes  that  the  crux 
of  the  problems  with  the  LFT  law  is  (i)  the  ambiguity  of 
the  law;  (2)  the  lack  of  a  clear  and  binding  LFT  policy 
directive;  (3)  the  reluctance  by  the  Services  to— for  fear  of 
a  stigma10 — ask  for  a  waiver  from  full-scale,  fiill-up  LFT  for 
those  programs  which  they  believe  LFT  to  be  unreasonably 
expensive  and  impractical;  and  (4)  the  absence  of  a  formal 
waiver  process  that  includes  a  procedure  for  identifying 
when  full-scale,  full-up  testing  is  or  is  not  unreasonably 


,0Th ere  is  also  the  possibility  that  the  Services  do  not  consider  it  appro¬ 
priate  for  Congress  to  become  involved  in  this  area  of  program  management 
and  acquisition. 
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expensive  and  impractical  which  would  eliminate  any  stigma 
associated  with  the  waiver. 
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The  Future  of  Vulnerability 
Assessment 


Characteristics  of  the  Future  Department  of  Defense 
Environment 

The  committee  recognizes  that  there  will  be  a  major 
change  in  the  environment  in  which  the  Department  of  De¬ 
fense  (DoD)  will  acquire  weapon  systems  in  the  future, 
given  the  dramatic  changes  in  the  world  socio-political  envi¬ 
ronment  and  the  current  difficulties  with  the  U.S.  economy. 
Almost  certainly,  the  future  DoD  budgets  for  acquisition  of 
new  systems  will  be  significantly  reduced,  with  a  concomi¬ 
tant  reduction  in  the  number  of  new  program  starts.  Two 
recent  examples  of  the  types  of  changes  to  be  expected  are 
the  reduction  of  B-2  aircraft  buy  from  75  to  20  and  the 
restructuring  of  the  RAH-66  COMANCHE  program  to  post¬ 
pone  production  while  emphasizing  and  stretching  out  the 
prototype  phase.  Future  budgets  for  improving  existing  sys¬ 
tems  most  likely  will  be  below — possibly  much  below — 
current  levels.  Future  budgets  for  the  technology  base  (not 
counting  independent  research  and  development)  are  diffi¬ 
cult  to  predict  at  this  time,  but  may  be  above  current  levels 
in  order  to  maintain  the  current  U.S.  technology  edge. 

In  spite  of  the  significant  reduction  in  the  future  DoD 
acquisition  budgets,  the  requirement  to  maintain  and  techno¬ 
logically  update  fielded  platforms  over  substantially  longer 
lifetimes  to  counter  new  threats  will  remain.  When  new 
programs  are  initiated  or  prototypes  moved  into  production, 
it  probably  will  be  in  response  to  important  new  threats  not 


able  to  be  defeated  by  current  platform  improvements.  In 
all  cases,  there  will  be  a  requirement  to  contain  the  costs 
of  the  new  or  prototype  system.  Although  program  cost 
containment  has  always  been  a  concern  within  DoD,  the 
committee  believes  that  minimizing  total  program  costs  in 
the  foreseeable  future  while  improving  weapon  system  abili¬ 
ties  to  counter  and  survive  new  threats  will  be  an  even  more 
important  issue. 

In  this  regard,  the  committee  notes  that  the  restructured 
program  for  prototyping  the  RAH-66  has  deferred  the  surviv¬ 
ability  and  live  fire  tests.  Because  there  is  no  intention  to 
go  into  production  at  this  time,  the  RAH-66  is  not  a  candidate 
for  Live  Fire  Testing.  The  committee  is  concerned  that  by 
deferring  the  survivability  and  live  fire  tests,  the  vulnerability 
of  the  prototype  design  will  not  receive  the  proper  attention. 
A  decision  could  be  made  in  the  future  to  move  the  prototype 
into  production,  with  no  further  vulnerability  assessments 
planned  to  determine  any  design  weaknesses.  However,  once 
this  decision  is  made,  the  helicopter  becomes  a  candidate 
for  Live  Fire  Testing.  Any  attempts  to  change  the  prototype 
design  to  reduce  vulnerability  once  the  decision  has  been 
made  to  go  into  production  may  be  met  with  much  resistance. 
This  is  going  to  create  an  adversarial  situation  again.  It 
would  be  much  better  to  find  and  correct  any  vulnerabilities 
in  the  design  during  development  of  the  prototype  than 
attempt  to  do  so  after  the  development  is  completed. 

In  addition  to  individual  program  cost  containment  issues. 
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the  committee  anticipates  a  reduction  in  the  test  and  evalua¬ 
tion  infrastructure  within  the  Office  of  the  Secretary  of  De¬ 
fense  (OSD)  and  in  each  of  the  Services  as  the  total  DoD 
budget  declines.  In  particular,  the  independent  charters, 
staffs,  and  facilities  of  the  vulnerability  assessment  commu¬ 
nity  may  have  to  be  changed,  as  well  as  the  emphasis  cur¬ 
rently  given  to  the  analysis  and  test  communities.  The  com¬ 
mittee  believes  that  new  approaches  to  many  of  the  current 
DoD  weapon  acquisition  procedures  may  have  to  be  devel¬ 
oped  to  respond  to  the  new  fiscal  threat  environment,  and 
among  these  approaches  may  be  changes  to  the  procedures 
and  infrastructure  for  vulnerability  assessment. 

Crucial  Requirements  of  Future  Vulnerability 
Assessment  Procedures  and  Infrastructure 

The  committee  believes  that  it  is  vital  for  the  future 
effectiveness  of  U.S.  military  aircraft  that  any  changes  made 
to  the  vulnerability  assessment  procedures  and  infrastructure 
must  not  degrade  the  current  capabilities  for  vulnerability 
assessment  within  DoD.  Vulnerability  assessment,  which 
is  an  integral  part  of  designing  for  survivability,  will  increase 
in  importance  as  the  numbers  of  front  line  aircraft  decrease 
and  the  lifetime  of  each  aircraft  is  extended.  Furthermore, 
because  fewer  new  systems  will  be  developed  in  the  future, 
the  vulnerability  of  each  of  these  new  systems  becomes 
more  important.  Consequently,  more  funding  for  vulnerabil¬ 
ity  assessment  may  be  required  for  the  few  systems  that  are 
developed,  rather  than  less  funding.1  On  the  positive  side 
of  this  situation,  as  more  new  starts  are  prototyped  and 
development  time  is  increased,  there  are  more  opportunities 
for  better  assessments.  For  example,  the  results  from  live 
fire  tests  on  full-scale,  full-up  targets  may  be  obtained  suffi¬ 
ciently  early  in  the  stretched-out  development  time  to  influ¬ 
ence  the  final  design.  An  additional  impetus  for  vulnerability 
assessment  is  the  close  and  important  connection  between 
combat  vulnerability  and  flight  safety.  Aircraft  designed  to 
take  hits  in  combat  and  to  survive  crashes  are  inherently 
safer  aircraft  for  the  aircrew  to  fly  in.  Many  design  features 
included  to  reduce  vulnerability,  such  as  fire  and  explosion 
suppression,  flight  control  reconfigurability,  hydraulic  power 
redundancy  and  separation,  rotor  blade  toughness,  and  en¬ 
gine  hardness,  also  prevent  the  loss  of  an  aircraft  due  to 
peacetime  malfunctions. 

Based  on  the  suppositions  given  above,  the  committee 
believes  that  the  vulnerability  assessment  capabilities  of  the 
future,  both  analytical  and  testing,  should  be  developed  as 
follows: 

1.  Vulnerability  assessment  will  continue  to  be  required 


'Even  though  the  funding  for  each  system  may  be  increased,  the  total 
expenditure  for  vulnerability  assessment  of  all  new  systems  may  be  less  than 
the  current  level  due  to  the  smaller  number  of  new  systems  in  development. 
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to  design  and  validate  the  vulnerability  expectations  of  new 
military  platforms.  Consequently,  the  current  methodologies 
must  be  improved.  However,  the  assessments  and  their  im¬ 
provements  must  use  the  available  funds  in  the  most  efficient 
way. 

2.  There  will  also  have  to  be  a  cost  reduction  within  the 
DoD  vulnerability  analysis  and  test  infrastructure,  because 
there  will  be  fewer  aircraft  to  assess,  while  still  maintaining 
necessary  staff  expertise  and  test  facilities  for  the  remaining 
programs. 

3.  A  vulnerability  test  and  evaluation  procedure  must  be 
developed  that  provides,  despite  any  cost  reductions,  a 
greater  level  of  trust  among  the  three  Services,  the  OSD. 
and  Congress  than  exists  today. 

The  committee’s  view  is  that  unless  a  procedure  for  vul¬ 
nerability  assessment  and  design  validation  can  be  developed 
that  increases  the  mutual  trust  among  participants,  there  is 
very  little  chance  that  the  requirement  to  reduce  the  funds 
expended  for  assessment  while  maintaining  the  capability 
to  produce  a  less  vulnerable  and  more  survivable  aircraft 
will  be  satisfied.  This  procedure  must  be  a  logically  based 
method  that  removes  all  emotionalism  and  arbitrariness  from 
the  assessment  process. 


Categories  for  Cost  Reduction  in  Vulnerability 
Assessment  While  Maintaining  or  Improving  the 
Current  Assessment  Capabilities 

Category  1 — Increased  Reliance  on  Analysis! Model¬ 
ing.  One  method  for  reducing  the  costs  of  vulnerability 
assessment  would  be  to  rely  more  on  the  analysis/modeling 
methodology.2  Based  on  its  own  review,  the  committee  be¬ 
lieves  that  there  appears  to  be  a  sufficient  start  of  a  modeling 
capability  and  weapons  effects  and  materials  data  base  to 
warrant  an  increased  dependence  on  analysis/modeling  for 
future  vulnerability  assessments  as  an  aid  in  design.  How¬ 
ever,  the  committee  also  believes  that  the  current  analytical 
methodology  and  supporting  data  bases  are  not  yet  suffi¬ 
ciently  robust,  correct,  precise,  and  representative  to  permit 
a  total  dependence  on  this  methodology.  Much  work  needs 
to  be  accomplished  in  the  model  development  and  in  the 
accumulation  of  weapons  effects  and  material  data  bases. 
Consequently,  much  of  live  fire  testing  in  the  future  should 
be  oriented  toward  verifying  the  improved  modeling  proce¬ 
dures,  extending  the  data  base  of  weapons  effects  and  mate¬ 
rial  responses,  and  validating  proposed  design  features  and 
equipment  for  reducing  vulnerability. 


:Relying  more  on  analysis  does  not  mean  that  the  full-scale,  full -up 
tests  mandated  by  Congress  should  be  discontinued.  Instead,  as  the  models 
axe  improved,  the  number  of  discovered  weaknesses  should  decrease. 
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Category  2 — Request  a  Waiver  from  the  Live  Fire  Test 
(LFT)  Law.  The  major  factor  in  the  cost  of  vulnerability 
assessment,  which  obviously  provides  the  major  opportunity 
for  cost  reduction,  is  the  requirement  for  the  full-scale,  full- 
up  Live  Fire  Test  (LFT)  program  mandated  by  the  LFT  law. 
This  law  was  written  to  prevent  the  neglect  of  vulnerability 
in  the  system  design  and  was  the  result  of  the  distrust  among 
the  Congress,  OSD,  and  the  Services  examined  in  Chapter 
4.  The  law  offers  a  waiver  from  the  Live  Fire  Tests.  If  this 
waiver  is  granted,  considerable  funds  would  be  freed  up  to 
be  used  in  vulnerability  analyses  and  Live  Fire  Tests  at 
the  sub-scale  level.  Under  the  current  Live  Fire  Test  and 
Evaluation  (LFT&E)  Guidelines,  there  is  no  guidance  as  to 
what  constitutes  an  unreasonably  expensive  and  impractical 
Live  Fire  Test  program  and  no  instructions  on  which  facts 
should  be  gathered  together  and  used  to  make  an  unbiased, 
impartial,  and  logical  waiver  decision.  Up  to  the  present 
time,  the  Services  have  not  applied  for  a  waiver  for  any 
program,  apparently  because  of  the  belief  that  their  LFT&E 
programs  were  in  compliance  with  the  law  and  because  they 
have  been  reluctant  to  take  advantage  of  the  waiver  for  fear 
of  a  stigma  attaching  itself  to  their  program. 

Many  people,  both  in  and  out  of  the  Services,  are  of 
the  opinion  that  full-scale,  full-up  Live  Fire  Testing  is 
unreasonably  expensive  and  impractical  for  all  aircraft,  that 
there  are  more  cost-effective  ways  to  obtain  a  design  with 
reduced  vulnerability,  and  that  this  conclusion  should  be 
obvious  to  any  one  who  has  thought  about  it  Among  the 
reasons  given  for  this  conclusion  are  the  facts  that  the 
results  come  too  late  in  the  development  cycle  to  influence 
the  design,  that  expensive  targets  are  at  risk  on  every  shot 
that  not  enough  data  are  obtained  for  statistical  validity, 
that  the  wrong  conclusion  might  be  reached,  and  all  the  other 
disadvantages  associated  with  Live  Fire  Testing  presented  in 
Chapter  2. 

Others  believe  that  full-scale  LFT  is  unreasonably  expen¬ 
sive  and  impractical  only  for  some  aircraft  such  as  large 
nontactical  aircraft  that  are  not  likely  to  encounter  threats 
to  their  survival,  and  is  not  unreasonably  expensive  and 
impractical  for  other  aircraft  such  as  small  tactical  aircraft 
that  are  very  likely  to  encounter  threats  to  their  survival; 
there  are  some  who  feel  that  LFT  is  not  unreasonably  expen¬ 
sive  and  impractical  for  any  aircraft  They  believe  that  only 
by  testing  the  full-scale,  full-up  aircraft  can  certain  design 
weaknesses  be  discovered,  that  it  is  never  too  late  to  change 
a  design  if  it  is  inadequate,  that  a  test  schedule  can  be 
designed  so  that  the  full-scale  target  is  not  at  risk  for  all 
shots,  and  all  the  other  advantages  associated  with  Live  Fire 
Testing  presented  in  Chapter  2.  All  of  these  opinions  and 
beliefs  on  both  sides  of  the  fence  are  subjective;  none  are 
objective.  No  quantification  of  the  benefits  and  risks  associ¬ 
ated  with  the  combination  of  analyses  and  full-scale,  full- 
up  live  fire  tests  compared  to  the  benefits  and  risks  associated 


57 

with  the  combination  of  analyses  and  only  sub-scale  live 
fire  testing  has  been  attempted.  Consequently,  a  decision  to 
grant  a  waiver  today  will  be  based  on  subjective  arguments. 

In  the  future,  a  formal  analytical  procedure  must  be  estab¬ 
lished  for  gathering  the  facts  necessary  to  determine  if  the 
full-scale,  full-up  Live  Fire  Tests  are  unreasonably  expensive 
and  impractical  with  respect  to  the  critical  vulnerability  is¬ 
sues  and,  if  they  are,  what  other  assessments  should  be 
conducted  in  place  of  the  complete  system  tests.  With  such 
a  procedure  in  place,  requesting  and  receiving  a  waiver, 
which  should  allow  much  of  the  vulnerability  assessment 
budget  to  be  transferred  to  other  assessment  tasks,  will  be 
an  acceptable  procedure.  No  stigma  should  be  attached  to 
the  waiver  because  the  procedure  for  obtaining  a  waiver  is 
a  rational  one.  Arbitrariness  and  emotionalism  have  been 
reduced  or  eliminated  and  replaced  with  objectivity. 

Such  a  procedure,  referred  to  here  as  a  risk-benefit  assess¬ 
ment,  could  be  developed  by  using  the  principles  of  risk 
analysis.  Risk  analysis  is  a  procedure  that  has  been  developed 
for  projects  that  involve  large  capital  outlays,  significant 
new  technology,  uncertainty,  and  regulatory  issues,  such  as 
the  Live  Fire  Test  law  (Cooper  and  Chapman,  1987).  This 
risk-benefit  methodology  will  formalize  and  standardize  the 
procedure  for  deciding  if  a  waiver  should  be  granted.  A 
risk-benefit  assessment  would  identify  and  quantify  the  risks 
and  benefits  associated  with  both  conducting  and  not  con¬ 
ducting  full-scale,  full-up  testing.  For  example,  the  benefits 
(e.g.,  a  reduction  in  vulnerability)  associated  with  full-scale, 
full-up  testing  of  a  relatively  inexpensive  combat  aircraft, 
many  of  which  are  very  likely  to  be  hit  in  combat,  may 
outweigh  the  benefits  (e.g.,  a  reduction  in  expenditures)  of 
not  testing  the  full-scale,  full-up  aircraft  Not  testing  the 
full-scale,  full-up  aircraft  puts  too  many  aircraft  at  an  unac¬ 
ceptable  risk  of  destruction.  On  the  other  hand,  the  benefits 
associated  with  such  tests  on  a  relatively  large  support  air¬ 
craft,  very  few  of  which  will  likely  be  hit  in  combat,  may  not 
outweigh  the  risks  of  not  conducting  the  tests.  The  committee 
believes  that  such  a  methodology  is  essential  to  the  process 
of  requesting  a  waiver.  If  one  is  not  developed,  the  arbitrary 
granting  of  a  waiver  will  continue  to  be  subject  to  consider¬ 
able  controversy,  and  attempts  may  be  made  to  avoid  all 
live  fire  testing. 

Category  3 — Consolidation  of  the  Vulnerability  Assess¬ 
ment  Infrastructure.  The  third  category  for  reducing  costs 
is  to  consolidate  the  various  Service  live  fire  test  facilities 
and  vulnerability  analysis/modeling  organizations.  Although 
the  committee  did  not  review  in  detail  this  aspect  of  the 
vulnerability  assessment  activities  and  capabilities  in  each 
of  the  three  Services,  it  believes  that  until  the  recent  DoD 
budget  downturn,  there  were  sufficient  programs  to  warrant 
the  continuation  of  more-or-less  similar  Service  live  fire  test 
capabilities.  Continuous  review  within  the  Services  and  by 
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OSD  has  already  led  to  reduction  of  duplicative  underutilized 
capabilities.  However,  in  the  decades  ahead,  the  expected 
requirements  for  the  vulnerability  testing  of  new  Service 
equipment  will  probably  fall  below  that  level  where  a  critical 
mass  of  broad-based  facilities  and  knowledgeable  staff  can 
be  maintained  within  any  of  the  individual  Services.  If  con¬ 
siderable  cost  savings  could  be  achieved  by  consolidating 
the  capabilities  of  these  facilities,  some  additional  form  of 
consolidation  beyond  that  currently  contemplated  should  be 
considered.  Furthermore,  an  overall  reduction  in  the  cost  of 
test  facilities  does  not  automatically  imply  a  reduction  in 


capabilities  at  every  facility.  In  fact,  by  downsizing  or  elimi¬ 
nating  some  facilities,  others  can  increase  in  capabilities. 
Perhaps  a  national  vulnerability  test  center  could  be  created 
that  would  serve  all  Services,  with  significantly  more  capa¬ 
bilities  than  currently  exist  at  any  one  facility  today,  at  a 
cost  below  that  in  effect  today. 
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Conclusions 

After  reviewing  the  vulnerability  assessment  methodolo¬ 
gies,  evaluating  the  cost,  effectiveness,  and  deficiencies  of 
these  methodologies,  and  reviewing  the  Live  Fire  Test  (LFT) 
law  and  the  Department  of  Defense  (DoD)  Live  Fire  Test 
&  Evaluation  (LFT&E)  program,  the  committee  has  come 
to  the  following  conclusions. 

•  Conclusions  Regarding  the  Live  Fire  Test  Law  <& 
the  DoD  LFT <5cE  Programs 

1.  The  committee  believes  that  the  requirements  in 
the  Live  Fire  Test  law  have  been  interpreted  in  several 
ways  and  that  these  different  interpretations  have  caused 
confusion  and  tension  in  the  Live  Fire  Test  programs. 
Nevertheless,  the  committee  believes  that  the  law  is  a 
valuable  contribution  to  vulnerability  assessment  and 
to  the  design  of  survivabie  aircraft  Furthermore,  it  is 
satisfactory  in  its  present  form  because  of  the  waiver 
process.  The  committee  believes  that  verification  of  vulner¬ 
ability  by  live  fire  testing  is  necessary  and  that  this  law 
ensures  that  verification. 

2.  The  committee  believes  that  the  1987  congressional 
Live  Fire  Test  law  mandates  live  fire  testing  of  full-scale, 
full-up  aircraft,  including  on-board  ordnance,  unless  a 
waiver  is  granted  by  the  Secretary  of  Defense.  Therefore, 
any  LFT&E  program  that  has  not  received  a  waiver  must 
conduct  full-scale,  full-up  tests.  This  law  was  written  be¬ 
cause  of  Congress’s  belief  that  the  Services  were  reluctant 
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to  fully  test  the  vulnerability  of  their  systems  as  they  were 
being  developed.  The  program  that  evoked  the  law  was 
the  Army’s  Bradley  Armored  Fighting  Vehicle  (AFV).  The 
AFVs  were  being  purchased  before  their  vulnerability  was 
fully  known.  Because  of  Congress’s  concern  that  a  similar 
situation  may  exist  for  systems  other  than  armored  vehicles, 
it  made  the  law  applicable  to  all  covered  systems,  including 
aircraft.  According  to  the  fiscal  year  (FY)  1988-1989  DoD 
Authorization  Act  Conference  report,  “The  conferees  intend 
that  the  Secretary  of  Defense  implement  this  section  (2366) 
in  a  manner  which  encourages  the  conduct  of  full-up  vulnera¬ 
bility  and  lethality  tests  under  realistic  combat  conditions, 
first  at  the  sub-scale  level  as  sub-scale  systems  are  developed, 
and  later  at  the  full-scale  level  mandated  in  the  legislation” 
(U.S.  Congress,  1988). 

3.  The  committee  believes  that  the  1988  Live  Fire  Test 
&  Evaluation  Guidelines  and  the  1989  Live  Fire  Test  & 
Evaluation  Planning  Guide  are  not  consistent  with  its 
interpretation  of  the  LFT  law.  The  Navy  and  the  Air  Force 
have  interpreted  the  1988  LFT&E  Guidelines  to  imply  that 
full-scale,  full-up  tests  are  not  required:  Furthermore,  the 
LFT&E  policies  presented  to  the  committee  do  not  consider 
such  tests  to  be  cost-effective,  particularly  if  on-board  ord¬ 
nance  is  included.  Consequently,  neither  Service  has  devel¬ 
oped  LFT&E  programs  that  contain  full-scale,  fiiU-up  Live 
Fire  Tests.  However,  both  Services  strongly  support  the 
conduct  of  sub-scale  inert  and  full-up  tests  throughout  the 
development  process.  The  Army  policy  on  LFT&E  supports 
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a  "building-block”  approach  consisting  of  component  testing 
through  full-scale,  full-up  system  testing  that  satisfies  the 
Live  Fire  Test  law.  The  Army  also  strongly  supports  sub¬ 
scale  testing.  The  emphasis  of  its  LFT&E  program  is  on 
sub-scale  testing,  with  limited  full-scale,  full-up  testing  to 
confirm  the  results  obtained  from  the  sub-scale  testing.  How¬ 
ever,  the  Army  LFT&E  program  for  the*  RAH-66  did  not 
contain  firm  plans  for  testing  a  full-scale,  full-up  helicopter 
the  full-scale  testing  was  going  to  be  conducted  only  if  the 
sub-scale  test  results  showed  it  to  be  necessary. 

4.  Because  all  three  Services  believe  that  an  LFT&E 
program  plan  that  contains  only  sub-scale  testing  is  in 
compliance  with  the  law  as  interpreted  by  the  Office  of 
the  Secretary  of  Defense  (OSD)  1988  LFT&E  Guidelines, 
no  waivers  have  been  requested.  The  OSD  Live  Fire  Test 
Office  has  been  unable  to  convince  them  that  an  LFT&E 
program  that  does  not  contain  full-scale,  full-up  tests  is  not 
in  compliance  with  the  law.  This  conflict  in  interpretation 
is  exacerbated  by  the  fact  that  all  of  the  current  Service 
aircraft  acquisition  programs  were  under  way  at  the  time 
the  law  was  written.  Furthermore,  the  1987  law  made  no 
provisions  for  funding  these  tests.  It  is  difficult  to  make  a 
major  change,  such  as  that  required  by  the  LFT  law,  in  the 
middle  of  a  test  and  evaluation  (T&E)  program  without 
additional  funding  and  schedule  delays. 

5.  The  committee  believes  that  a  waiver  is  required 
to  omit  the  full-scale,  full-up  tests.  Congress  recognized 
that  there  may  be  weapon  systems  for  which  a  full-scale, 
full-up  test  program  is  unreasonably  expensive  and  impracti¬ 
cal  when  it  wrote  the  LFT  law.  Therefore,  it  included  a 
provision  for  the  Secretary  of  Defense  to  grant  a  waiver 
from  such  tests,  provided  a  plan  for  alternatives  to  realistic 
vulnerability  testing  is  prepared. 

6.  The  committee  believes  that  there  are  aircraft  for 
which  a  full-scale,  full-up  test  program  is  unreasonably 
expensive  and  impractical,  and  that  there  are  aircraft 
for  which  a  full-scale,  full-up  test  program  is  neither 
unreasonably  expensive  nor  impractical.  Thus,  there  are 
programs  for  which  a  waiver  is  justified  and  programs  for 
which  a  waiver  is  not  justified.  The  committee  also  believes 
that  in  order  to  make  the  waiver  process  a  viable  alternative 
LFT  path,  the  waiver  process  must  be  formalized.  This  for¬ 
mal  process  must  contain  a  procedure  that  can  identify  when 
the  full-scale,  fiill-up  tests  are  unreasonably  expensive  and 
impractical,  and  when  they  are  not  This  formal  procedure 
would  remove  the  threat  of  a  stigma  being  associated  with 
a  waiver. 

7.  The  committee  believes  there  should  be  no  stigma 
attached  to  a  waiver  because  the  waiver  Is  an  acceptable 
alternative  LFT&E  path.  Apparently,  the  Services  are  op¬ 
posed  to  requesting  a  waiver  for  any  program  because  of 
the  apprehension  that  their  program  will  suffer  in  some 
manner  as  a  result  of  the  waiver.  They  believe  that  a  stigma 


will  be  associated  with  such  a  request.  The  Committee  holds 
the  opinion  that  the  waiver  is  an  acceptable  alternate  LFT&E 
path.  The  waiver  is  accepted  by  Congress  as  reasonable 
when  the  full-scale,  full-up  tests  are  certified  by  the  Secretary 
of  Defense  to  be  unreasonably  expensive  and  impractical, 
and  an  alternative  plan  for  realistic  vulnerability  testing  is 
proposed.  No  approval  by  Congress  is  necessary  if  the  certi¬ 
fication  is  given  before  the  system  enters  full-scale  engi¬ 
neering  development. 

8.  A  serious  problem  in  both  the  analyses  and  the 
Joint  Live  Fire  Testing  (JLF)  of  aircraft  has  been  the 
omission  of  on-board  ordnance  as  a  critical  component. 
It  may  be  one  of  the  largest  contributors  to  vulnerability, 
particularly  for  aircraft  that  carry  the  ordnance  internally. 
Alternatively,  on-board  ordnance  may  reduce  the  aircraft’s 
vulnerability  by  shielding  critical  components.  Analysis  and 
testing  must  be  conducted  both  with  and  without  on-board 
ordnance  in  order  to  properly  account  for  this  materiel. 

9.  The  stated  intent  of  the  full-scale,  full-up  tests  man¬ 
dated  by  the  Live  Fire  Test  law  is  to  aid  in  design  by 
providing  information  on  any  weaknesses  sufficiently 
early  in  the  design  process  to  allow  the  weaknesses  to  be 
corrected.  However,  the  Services  and  industry  believe  that 
the  full-scale  LFTs  are  conducted  too  late  in  the  development 
cycle  to  have  any  impact  on  the  design.  The  committee 
believes  that  if  no  major  vulnerabilities  are  discovered  in 
the  full-scale  tests,  this  information  is  of  great  value  to  the 
acquisition  decision  makers,  and  if  a  major  vulnerability  is 
discovered,  it  should  be  corrected.  Other  arguments  against 
the  full-scale,  full-up  tests  are  the  facts  that  full-scale  tests 
may  be  conducted  on  a  nonrepresentative  target  and  consume 
money  that  could  be  used  for  more  of  the  earlier  sub-scale 
tests.  Counterarguments  are  that  there  is  much  to  be  learned 
from  testing  full-scale  targets  similar  to  the  complete  system 
and  that  sufficient  funds  need  to  be  programmed  for  vulnera¬ 
bility  testing.  Vulnerability  testing  is  an  important  T&E 
task  in  the  acquisition  process  that  has  been  significantly 
underfunded  in  the  past  The  committee  believes  there  is  a 
place  for  full-scale  testing  somewhere  in  the  life  of  the 
aircraft.  For  those  aircraft  in  which  full-scale,  full-up  testing 
is  unreasonably  expensive  and  impractical  during  the  full- 
scale  development  phase,  later  full-up  tests  on  production 
aircraft  that  are  no  longer  operational,  such  as  done  in  the 
JLF  program,  can  impact  any  subsequent  modifications  of 
the  aircraft,  as  well  as  future  aircraft  designs. 

10.  The  implied  intent  of  the  Live  Fire  Test  law  is  to 
force  the  consideration  of  vulnerability  during  the  design 
process.  In  the  programs  the  committee  examined,  evidence 
of  early  considerations  of  vulnerability  was  obvious.  Thus, 
even  though  full-scale  live  fire  tests  had  not  been  planned  or 
conducted,  the  law  has  had  beneficial  effects.  The  committee 
believes  that  additional  motivation  to  consider  vulnerability 
in  the  design  can  be  obtained  by  placing  realistic  design 
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requirements  on  the  maximum  amount  of  vulnerability  al¬ 
lowed  at  program  inception.  This  requirement  on  the  design, 
coupled  with  appropriate  live  fire  testing  and  the  Live  Fire 
Test  law,  would  better  meet  the  intent  of  the  LFT  law  and 
good  DoD  design  practice. 

11.  The  lack  of  a  definition  of  the  specific  threat  muni¬ 
tions  to  be  used  in  design  and  in  Live  Fire  Testing  has  re¬ 
sulted  in  considerable  controversy  regarding  which 
threat  weapons  to  use  in  the  Services’  LFT  programs. 
Even  the  cursory  review  of  threats  posed  to  the  systems  exam¬ 
ined  by  the  committee  leads  to  the  conclusion  that  the  design 
threats  are  probably  not  the  only  threats  likely  to  be  encoun¬ 
tered  in  combat.  Furthermore,  other  threats  likely  to  be  en¬ 
countered  in  combat  may  be  more  lethal  than  the  design 
threats.  This  is  particularly  true  for  the  C-17,  and  possibly  the 
RAH-66.  Nevertheless,  the  acquisition  process  must  include 
a  threat  projection  and  a  design  threat  selection  as  an  integral 
feature;  it  is  both  necessary  and  feasible.  Without  it  there  is 
no  real  discipline  in  the  development  process,  and  the  testing 
process  is  free  to  test  against  whatever  threat  it  chooses,  rele¬ 
vant  or  not  relevant.  The  ambiguity  in  the  phrase  “munitions 
likely  to  be  encountered  in  combat”  makes  it  possible  to  put 
a  system  in  an  unfavorable  position  based  on  Live  Fire  Tests 
against  threats  for  which  the  system  was  not  designed.1  The 
committee  believes  that  the  design  threat  selected  for  some 
systems  is  not  the  major  threat  likely  to  be  encountered  when 
these  systems  are  fielded.  The  design  threat  must  be  projected 
forward  in  time  in  order  to  prevent  the  system  capabilities 
from  falling  behind  the  threat  capabilities.  There  will  be  an 
understandable  reluctance  on  the  part  of  the  intelligence  com¬ 
munity  to  make  such  a  projection,  but  it  can,  and  must,  be 
done.  Furthermore,  the  design  threat  must  be  reviewed  at 
each  milestone  or  other  major  decision  point. 

12.  Apparent  separation  of  the  oversight  of  vulnerabil¬ 
ity  analysis  from  the  oversight  of  live  fire  testing,  both 
of  which  are  part  of  the  T&E  process,  has  created  a 
situation  that  is  detrimental  to  the  overall  OSD  vulnera¬ 
bility  program.  The  committee  is  concerned  that  the  appar¬ 
ent  organizational  separation  of  OSD  review  of  vulnerability 
analyses  and  Live  Fire  Testing  that  currently  exists  could 
substantially  impede  a  coordinated  program  to  determine 
vulnerability  policy,  facility  requirements,  and  model  and 
data  base  development  The  problem  with  the  separation 
of  the  two  oversight  responsibilities  is  that  there  can  be 
undue  emphasis  placed  on  one  or  the  other  methodologies. 
By  making  one  office  responsible  for  both,  a  proper  sense 
of  perspective  and  a  synergistic,  long-term  development  pro¬ 
gram  can  be  achieved.  Furthermore,  the  committee  believes 
that  oversight  to  the  analysis/modeling  methodology  is  a 


'The  Army's  DIVAD  gun  system  was  a  victim  of  this  practice.  It 
met  the  documented  target  requirements  but  failed  against  nondocumented 
targets. 
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T&E  issue.  The  committee  believes  that  the  separation  may 
be  a  major  contributor  of  the  current  problem.  By  combining 
the  oversight  of  the  two  methodologies,  the  proper  emphasis 
can  be  given  to  each  methodology.  The  committee  believes 
that  in  the  future  DoD  environment  of  prototypes  and  de¬ 
ferred  production,  overall  vulnerability  reduction  in  the  de¬ 
sign  can  best  be  served  by  the  integration  of  analyses  and 
supporting  live  fire  tests. 

•  Conclusions  Regarding  the  Vulnerability  Assess- 
ment  Methodologies 

13.  Based  upon  its  review  of  the  two  methodologies,  the 
committee  concludes  that  both  vulnerability  analysis  and 
live  fire  testing,  including  the  mandated  Live  Fire  Testing, 
are  essential  in  a  mix  peculiar  to  each  aircraft  develop¬ 
ment  program.  The  committee  believes  that  a  primary  appli¬ 
cation  for  these  methodologies  should  be  to  aid  in  the  design 
of  aircraft  throughout  the  development  process.  The  proper 
design  and  validation  of  the  vulnerability  of  an  aircraft  re¬ 
quire  a  well-planned  application  of  both  methodologies,  in¬ 
cluding  analyses,  sub-scale  testing,  and  full-scale  testing.  The 
importance  of  early  sub-scale  testing  to  the  design  cannot  be 
overemphasized.  The  analytical  and  testing  aspects  of  vulner¬ 
ability  design  and  assessment  must  be  not  conducted  inde¬ 
pendently.  A  consistent  oversight  of  the  entire  process  is  re¬ 
quired.  In  general,  analysis/modeling  is  all  that  is  available 
in  the  very  early  design  stages,  whereas  confirming  sub-scale 
testing  is  essential  in  the  middle  and  later  design  stages.  The 
sub-scale  tests  also  provide  information  for  the  databases  that 
support  the  analysis/modeling  efforts.  Full-scale  testing,  be¬ 
cause  it  occurs  late  in  the  development  cycle,  is  used  to  dis¬ 
cover  any  weaknesses  of  the  total  and  integrated  design. 

14.  The  committee  believes  that  both  methodologies 
need  to  be  improved  and  that  these  improvements  should 
be  mutually  beneficial.  There  appears  to  be  a  sufficient 
start  of  a  modeling  capability  and  weapons  effects  and  mate¬ 
rials  data  base  to  warrant  an  increased  dependence  on  analy¬ 
sis/modeling  for  future  vulnerability  assessments  as  an  aid 
in  design.  However,  the  committee  also  believes  that  the 
current  analytical  methodology  and  supporting  data  bases 
are  not  yet  sufficiently  robust,  correct,  precise,  representa¬ 
tive,  and  interactive  to  permit  a  total  dependence  on  this 
methodology.  Much  work  needs  to  be  accomplished  in  the 
model  development  and  in  the  accumulation  of  weapons 
effects  and  material  P**  data  bases.  Consequently,  live  fire 
testing  in  the  future  should  be  oriented  toward  verifying  the 
improved  modeling  procedures,  extending  the  data  base  of 
weapons  effects  and  material  responses,  and  validating  pro¬ 
posed  design  features  and  equipment  for  reducing  vulnera¬ 
bility.  The  analysis/modeling  methodology  requires  addi¬ 
tional  support  to  continue  the  development  of  models  that 
account  for  all  of  the  phenomena  and  damage  effects  ob¬ 
served  in  live  fire  tests  and  in  combat.  In  particular,  addi¬ 
tional  realistic  sub-scale  testing,  both  inert  and  full-up. 


I 


62 

is  necessary  in  order  to  continue  the  developmer.:  of  the 
P**  data  base  needed  for  improved  models.  This  requires 
the  development  or  improvement  of  test  facilities  that  can 
perform  such  tests. 

•  Conclusions  Regarding  (he  Vulnerability  Programs 
for  Aircraft 

15.  The  vulnerability  of  currently  fielded  U.S.  aircraft 
will  become  more  important  in  the  future.  Under  present 
funding  expectations,  current  aircraft  are  going  to  remain  in 
the  inventory  for  many  more  years.  These  aircraft  are  going 
to  require  product  improvements  because  of  the  anticipated 
improvements  in  the  weapons  available  to  the  Third  World. 
One  of  these  product  improvements  should  be  in  the  area 
of  vulnerability  reduction.  No  formal  process  currently  exists 
to  focus  routinely  on  changes  in  the  vulnerability  of  U.S. 
aircraft  caused  by  the  increase  in  weapon  lethality.  Such  a 
routine  vulnerability  reduction  review  should  be  established. 
Vulnerability  reduction  as  a  means  of  achieving  survivability 
enhancement  is  particularly  important  for  existing  aircraft 
that  cannot  take  advantage  of  the  new  stealth  technology. 

16.  There  is  insufficient  attention  given  to  the  require¬ 
ment  to  design  for  vulnerability.  DoD  Instruction  (DoDI) 
5000.2  includes  survivability  as  a  critical  system  characteris¬ 
tic  and  requires  that  survivability  objectives  be  defined  ini¬ 
tially  at  Milestone  I  and  finally  at  Milestone  II.  However,  no 
specific  reference  to  vulnerability  is  made  in  DoDI  5000.2. 
Vulnerability  requirements  should  be  identified  as  part  of 
the  survivability  characteristics  and  incorporated  in  develop¬ 
ment  contracts. 

17.  The  collection  of  actual  combat  data  on  the  vulner¬ 
ability  of  U.S.  aircraft  is  not  given  proper  emphasis. 
Peacetime  live  fire  testing,  as  well  as  computer-based  model¬ 
ing,  would  benefit  greatly  from  a  comparison  with  actual 
combat  data.  However,  the  procedures  required  to  collect 
the  proper  data  are  not  in  place.  In  briefings  provided  to  the 
committee,  a  list  of  lessons  learned  from  the  attempts  to 
collect  combat  survivability  data  during  Desert  Storm,  in¬ 
cluding  the  following:  (1)  existing  official  reporting  systems 
were  not  adequate  for  capturing  survivability  data;  (2)  valu¬ 
able  records  were  destroyed  because  of  established  retention 
limits;  (3)  data  questionnaires  could  not  be  completed  ade¬ 
quately  by  field  personnel  on  their  own,  (4)  permission  for 
data  collectors  to  enter  the  theater  was  strongly  resisted;  and 
(5)  arrangements  to  support  the  data  collectors  were  not  in 
place  and  were  worked  out  with  great  difficulty  for  the  few 
collection  teams  that  did  deploy. 

•  Conclusions  Regarding  the  Vulnerability  Infrastructure 

18.  The  process  of  designing  and  testing  for  vulnerabil¬ 
ity  is  extremely  complex  and  would  benefit  from  continu¬ 
ous  input  and  oversight  from  a  broad  range  of  experts 
in  the  vulnerability  community.  It  is  important  that  cooper¬ 
ation  be  established  among  the  Services  and  between  the 
Services  and  the  Live  Fire  Test  Office.  The  Joint  Technical 
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Coordinating  Group  on  Aircraft  Survivability  (JTCG/AS)  is 
an  organization  that  has  fostered  this  type  of  teamwork  and 
inter-Service  cooperation.  However,  it  would  be  useful  to 
have  a  standing  board  of  vulnerability  experts  annually  re¬ 
view  the  programs  and  plans  with  the  Director,  Test  and 
Evaluation. 

19.  The  vulnerability  community  of  the  future  most 
likely  will  become  smaller  in  both  the  number  of  pro¬ 
grams  and  the  size  of  the  infrastructure.  The  committee 
recognizes  the  fact  that  the  Department  of  Defense  is  going 
to  reduce  the  size,  funding,  and  number  of  aircraft  programs, 
both  new  and  product  improvements.  There  most  likely  will 
be  a  corresponding  drawdown  in  the  related  vulnerability 
assessment  activities,  both  in  analyses/modeling  and  in  test¬ 
ing.  The  committee  believes  that  this  drawdown  should  be 
carried  out  very  carefully  to  ensure  that  essential  vulnerabil¬ 
ity  assessment  personnel,  capabilities,  and  facilities  are  not 
lost  in  the  process. 

Recommendations 

Based  upon  the  results  of  the  committee’s  study  and  the 
conclusions  given  above,  the  committee  makes  the  following 
recommendations: 

•  Recommendations  Regarding  the  DoD  Live  Fire  Test 
&  Evaluation  Program 

1.  The  committee  recommends  that  the  Director,  Test 
and  Evaluation,  issue  Guidelines  that  replace  the  1988 
Live  Fire  Test  &  Evaluation  Guidelines  and  that  more 
clearly  conform  with  the  requirements  for  the  full-scale, 
full-up  tests  mandated  by  the  Live  Fire  Test  law.  The 
binding  force  of  the  existing  1988  LFT&E  Guidelines  is 
unclear  to  the  committee  and  to  the  Services,  and  should  be 
replaced  with  a  directive  whose  force  is  understood.  The 
recommended  directive  should  completely  define  the  proce¬ 
dures  and  requirements  for  planning  and  conducting  the  Live 
Fire  Test  and  Evaluation  program  for  both  sub-scale  and 
full-scale  tests.  The  directive  should  require  the  conduct  of 
vulnerability  tests  under  realistic  combat  conditions,  first  at 
the  sub-scale  level  as  sub-scale  systems  are  developed,  and 
later  at  the  full-scale  level  mandated  in  the  legislation.  In 
addition,  the  directive  should  contain  a  formal  process  for 
requesting  a  waiver  and  the  requirements  for  developing  the 
alternatives  to  the  realistic  survivability  testing  of  the  full- 
scale.  full-up  system. 

2.  The  committee  recommends  that  the  Director,  Test 
and  Evaluation,  formalize  the  waiver  process  by  devel¬ 
oping  a  risk-benefit  assessment  methodology  that  can  be 
used  uniformly  to  determine  whether  a  full-scale,  full-up 
test  program  for  any  particular  aircraft  is  “unreasonably 
expensive  and  impracticaL”  The  methodology  must  also 
be  applicable  to  the  evaluation  of  the  alternate  Live  Fire 
Test  program  for  the  sub-scale  targets.  The  process  for 
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requesting  a  waiver,  described  in  the  DoD  directive  recom¬ 
mended  above,  should  include  a  risk-benefit  assessment 
methodology  that  quantifies  the  benefits  associated  with  full- 
scale,  full-up  Live  Fire  Tests  and  the  risks  associated  with 
waiving  these  tests.  Such  a  methodology  should  give  empha¬ 
sis  to  early  testing  and,  where  possible,  consider  the  desir¬ 
ability  of  Live  Fire  Testing  of  full-scale  development  proto¬ 
types  and  structural  test  models.  Once  the  benefits  and  risks 
have  been  quantified,  a  decision  can  be  made  as  to  whether 
the  full-scale,  full-up  tests  are  unreasonably  expensive  and 
impractical.  The  committee  strongly  believes  that  such  a 
methodology  is  essential  to  the  process  of  requesting  a 
waiver. 

3.  The  committee  recommends  that  the  Secretary  of 
Defense  take  measures  to  ensure  (a)  that  the  LFT&E 
Guidelines  are  properly  enforced  by  requiring  either  that 
covered  systems  be  subjected  to  full-scale,  full-up  testing 
or  that  a  waiver  be  obtained;  (b)  that  any  waiver  be 
fully  justified;  (c)  that  the  waiver  process  be  uniformly 
applied;  and  (d)  that  no  stigma  be  attached  to  the  use 
of  the  waiver  process.  The  committee  believes  that  re¬ 
questing  a  waiver  is  a  legitimate  procedure  that  must  not 
adversely  affect  the  program.  The  granting  of  a  waiver  does 
not  eliminate  all  requirements  for  Live  Fire  Testing;  an 
acceptable  alternative  realistic  vulnerability  assessment  pro¬ 
gram  must  still  be  conducted.  Furthermore,  the  availability 
of  the  risk-benefit  methodology  in  recommendation  will 
remove  the  arbitrary  basis  for  granting  a  waiver  currently 
in  place  and  replace  it  with  a  logically  based  procedure  used 
for  other  large-scale  projects  in  which  risk  is  involved. 

4.  The  committee  recommends,  for  the  full-scale,  full- 
up  Live  Fire  Tests,  that  the  specific  “likely  to  be  encoun¬ 
tered”  munitions  referred  to  in  the  Live  Fire  Test  law 
be  the  weapon(s)  specified  in  the  requirements  documen¬ 
tation  for  the  system,  projected  forward  to  the  time  when 
the  system  is  to  be  fielded.  Furthermore,  the  threat  should 
be  reviewed  and  updated  periodically  at  the  milestone 
decision  points  to  ensure  that  the  specified  design  weap- 
on(s)  is  representative  of  the  major  “likely  to  be  encoun¬ 
tered9*  threat(s)  to  the  system.  There  has  been  considerable 
disagreement  on  what  “weapons  likely  to  be  encountered  in 
combat”  means  and  what  weapons  should  be  used  for  system 
design  and  in  the  Live  Fire  Tests.  The  design  weapon(s) 
specified  in  the  requirements  documentation  must  be  the 
best  estimate  of  the  primary  threat,  projected  forward  to  the 
time  the  system  is  to  be  fielded.  Selecting  threats  for  the 
design  that  are  less  lethal  than  others  likely  to  be  encountered 
is  unacceptable.  Furthermore,  this  design  threat  must  be  the 
threat  used  to  satisfy  the  system  tests  mandated  by  the  Live 
Fire  Test  law.  Without  this  linkage  between  a  realistic  design 
threat  and  the  test  threat,  the  test  agency  can  arbitrarily  select 
threats  that  may  not  meet  the  user’s  requirement  for  the 
system  and  that  may  jeopardize  the  future  of  the  program. 
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For  the  component  and  subsystem  tests  conducted  during 
the  design  phase,  the  committee  encourages  the  use  of  threats 
more  lethal  than  the  design  threat  when  appropriate. 

5.  The  committee  recommends  that  the  Director,  Test 
and  Evaluation,  expand  the  charter  of  the  Live  Fire  Test 
and  Evaluation  program  from  its  current  oversight  of 
those  tests  that  are  part  of  the  congressionally  mandated 
LFT  program  to  include  oversight  of  vulnerability  assess¬ 
ment.  This  new  OSD  program,  known  perhaps  as  the  Vul¬ 
nerability  Test  and  Evaluation  program,  would  have  broad 
oversight  of  the  evaluation  of  the  vulnerability  of  the  system 
design  throughout  the  lifecycle  of  the  system  and  would 
be  the  Services’  advocate  for  the  recommended  integrated 
vulnerability  evaluations  at  OSD  milestone  reviews.  These 
evaluations  would  be  accomplished  using  both  analyses  and 
live  fire  testing,  including  all  of  the  Live  Fire  Testing  man¬ 
dated  by  the  LFT  law. 

•  Recommendations  Regarding  the  Vulnerability  Assess¬ 
ment  Methodologies 

6.  The  committee  recommends  that  both  the  analysis 
community  and  the  live  fire  testing  community  routinely 
include  on-board  ordnance  in  their  assessments.  A  waiver 
to  allow  full-scale  Live  Fire  Tests  without  on-board  ordnance 
should  be  granted  only  after  an  examination  of  the  results 
from  alternate  live  fire  tests  of  sub-scale  components  and 
their  integration  into  analyses  of  the  full-up  aircraft  carrying 
such  ordnance. 

7.  The  committee  recommends  that  the  Secretary  of 
Defense  direct  the  multi-Service  coordinated  develop¬ 
ment  and  authorization  for  use  of  improved  analytical 
vulnerability  assessment  models  that  are  applicable  to 
all  military  aircraft  The  committee  believes  it  is  inevitable 
that  the  emphasis  given  to,  and  reliance  on,  the  models  will 
increase  in  the  future  as  budget  limitations  force  greater 
reliance  on  prototyping.  Consequently,  it  is  imperative  that 
the  models  be  improved.  The  current  JTCG/AS-approved 
models  could  form  the  basis  for  the  new  models.  The  1987 
General  Accounting  Office  (GAO)  study  on  Live  Fire  Test¬ 
ing  provides  many  suggestions  on  how  to  improve  these 
models. 

8.  The  committee  recommends  that  a  long-term  live 
fire  test  program  be  funded  in  which  realistic  compo¬ 
nents,  subsystems,  and  systems  are  specifically  tested  to 
develop  a  data  base  to  support  the  analytical  models. 
The  committee  believes  that  improvements  in  the  analyses/ 
models  recommended  above  can  be  achieved  only  when 
properly  supported  by  live  fire  testing  programs  and  phenom¬ 
enological  investigations,  and  the  committee  is  concerned 
that  the  present  OSD  Live  Fire  Test  priorities  do  not  ade¬ 
quately  support  this  data  base  improvement  The  funding 
for  these  tests  should  be  provided  by  the  Director,  Test  and 
Evaluation. 

9.  The  committee  recommends  that  the  Secretary  of 
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Defense  (a)  establish  a  program  to  examine  the  combat 
data  collected  from  Desert  Storm  for  “lessons  learned” 
regarding  the  susceptibility  and  vulnerability  of  U.S.  and 
allied  aircraft;  and  (b)  develop  formal,  institutionalized 
procedures  for  collecting  data  in  future  conflicts,  for 
ensuring  that  the  data  collectors  have  access  to  the  the¬ 
ater,  and  for  permanently  storing  the  data.  The  combat 
survivability  data  collection  program  should  reflect  the  im¬ 
portance  of  collecting  and  preserving  the  data  and  should 
be  coordinated  among  the  three  Services  through  a  joint 
agency,  such  as  the  JTCG/AS. 

*  Recommendations  Regarding  the  Vulnerability  Pro - 
grams  for  Aircraft 

10.  Because  of  the  expected  service  life  extension  of 
currently  fielded  U.S.  military  aircraft,  the  committee 
recommends  that  the  Under  Secretary  of  Defense,  Acqui¬ 
sition  establish  a  formal  vulnerability  assessment  and 
reduction  program  for  these  aircraft  This  program  should 
require  that  all  product  improvement  or  upgrade  programs 
to  existing  aircraft  include  vulnerability  reduction  as  a  major 
goal  of  the  program. 

1 1.  The  committee  recommends  (a)  that  a  vulnerabil¬ 
ity  assessment  program  be  an  integral  part  of  every  air¬ 
craft  acquisition  program;  (b)  that  vulnerability  assess¬ 
ment  and  evaluation  be  a  specific  item  examined  at  each 
formal  milestone  review;  and  (c)  that  adequate  funds  be 
appropriated  to  the  program.  The  specific  distribution  of 
the  funds  between  analysis/modeling  and  live  fire  testing 
for  each  program  should  be  proposed  by  the  individual  Serv¬ 
ice,  with  OSD  review  and  acceptance. 

12.  The  committee  recommends  that  aircraft  pro¬ 
grams  that  become  “prototype”  programs,  such  as  the 
RAH-66,  not  be  excluded  from  live  fire  testing.  The  RAH- 
66  COMANCHE  helicopter  has  recently  been  changed  to  a 
“prototype”  program.  The  committee  is  concerned  that  the 
RAH-66  might  be  developed  as  a  prototype  without  adequate 
consideration  or  testing  of  its  vulnerability.  If  the  decision 
is  made  at  a  later  date  to  go  into  production  with  the  proto¬ 
type,  it  will  be  too  late  to  conect  any  design  weaknesses. 

13.  The  committee  recommends  that  specific  vulnera¬ 
bility  requirements  on  the  design  be  a  part  of  the  surviv¬ 
ability  objectives  defined  at  Milestones  I  and  n.  These 
vulnerability  requirements  should  be  identified  as  part  of  the 
survivability  characteristics  and  incorporated  in  the  aircraft 
development  contracts. 

•  Recommendations  Regarding  the  Vulnerability  Infra¬ 
structure 

14.  The  committee  recommends  that  the  Director,  Test 
and  Evaluation,  establish  a  permanent  Senior  Vulnera¬ 
bility  Assessment  Board  comprised  of  senior  Services’ 


technical  leaders,  high-level  OSD  officials,  and  nationally 
recognized  experts  from  industry  and  academia.  This 
board  would  be  advisory  to  the  Director,  Test  and  Evalua¬ 
tion,  and  chartered  to  review  annually  the  proposed  vulnera¬ 
bility  assessment  and  budgets  of  DoD  and  to  review  the 
vulnerability  assessment  programs  on  specific  aircraft  pro¬ 
grams  as  the  need  arises.  This  board  would  be  similar  to 
the  boards  already  formed  for  conduct  of  coordinated  6.1, 
6.2,  and  6.3a  Tech  Base  programs  in  the  DoD.  The  commit¬ 
tee  believes  that  such  a  board  would  provide  the  Services 
with  a  “before-the-fact”  input  into  the  establishment  of  vul¬ 
nerability  policy  and  would  lead  to  a  better  acceptance  of 
this  policy. 

15.  The  committee  recommends  that  studies  be  con¬ 
ducted  to  determine  if  the  existing  Army,  Navy,  and  Air 
Force  vulnerability  analysis  community,  test  facilities, 
and  infrastructure  can  be  reduced  proportionally  to  the 
expected  overall  infrastructure  reduction  within  DoD. 
Project  Reliance,  the  existing  senior  joint  Services’  R&D 
cooperation  group,  should  be  charged  with  conducting  the 
studies  of  how  best  to  accomplish  a  meaningful  infrastruc¬ 
ture  reduction.  As  a  part  of  this  consolidation  study,  mecha¬ 
nisms  for  accommodating  unique  Service  needs  in  consoli¬ 
dated  testing  facilities  must  be  developed  in  order  to  allow 
multi-Service  acceptance  of  data  derived  from  singularly 
designated  facilities.  The  committee  believes  that  this  draw¬ 
down  should  be  carried  out  very  carefully  to  ensure  that 
essential  vulnerability  assessment  personnel,  capabilities, 
and  facilities  are  not  lost  in  the  process. 

The  Future 

The  committee  recommends  to  the  Secretary  of  De¬ 
fense  that  the  broad  issue  of  how  to  both  design  and  test 
for  vulnerability  in  an  austere  future  be  studied.  Present 
concepts  of  analyses  and  live  fire  testing  for  vulnerability 
may  not  be  adequate  in  a  future  of  reduced  budgets,  fewer 
fielded  aircraft,  fewer  program  starts,  smaller  procurement 
numbers,  and  more  "storage  on  the  shelf”  of  technology 
capabilities  with  less  time  to  react  to  emergencies.  When 
such  a  study  has  been  completed  and  an  effective  process 
has  been  developed  for  vulnerability  design  and  validation, 
OSD  should  consult  with  Congress  regarding  revisions  to 
the  LFT  law  that  reflect  this  new  process. 

Reference 

•  U.S.  Congress,  1988.  FY88-89  DoD  Authorization  Act  Confer¬ 
ence  Report,  Live-Fire  Testing  (Sec.  802). 
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The  Live  Fire  Test  Legislation 


FY87  DoD  Authorization  Act 

SEC.  910.  TESTING  OF  CERTAIN  WEAPON  SYS¬ 
TEMS  AND  MUNITIONS 

(a)  Survivability  and  Lethality  Testing  and  Operational 
Testing. 

— (1)  Chapter  139  of  title  10,  United  States  Code,  is 
amended  by  adding  after  section  2365  (as  added  by  sec¬ 
tion  909)  the  following  new  section: 

2366.  Major  systems  and  munitions  programs:  surviv¬ 
ability  and  lethality  testing;  operational  testing 
“(a)  Requirements — The  Secretary  of  Defense  shall  pro¬ 
vide  that — 

“(1)  a  covered  system  may  not  proceed  beyond  low- 
rate  initial  production  until  realistic  survivability  testing 
of  the  system  is  completed  in  accordance  with  this  section; 

“(2)  a  major  munition  program  or  a  missile  program 
may  not  proceed  beyond  low-rate  initial  production  until 
realistic  lethality  testing  of  the  program  is  completed  in 
accordance  with  this  section;  and 

“(3)  a  major  defense  acquisition  program  may  not 
proceed  beyond  low-rate  initial  production  until  initial 
operational  test  and  evaluation  of  the  program  is  com¬ 
pleted  in  accordance  with  this  section. 

“(b)  Test  Guidelines — 

“(1)  Survivability  and  lethality  tests  required  under 
subsection  (a)  shall  be  carried  out  sufficiently  early  in 


the  development  phase  of  the  system  or  program  to  allow 
any  design  deficiency  demonstrated  by  the  testing  to  be 
corrected  in  the  design  of  the  system,  munition,  or  missile 
before  proceeding  beyond  low-rate  initial  production. 

“(2)  In  the  case  of  a  major  defense  acquisition  pro¬ 
gram,  no  person  employed  by  the  contractor  for  the  system 
being  tested  may  be  involved  in  the  conduct  of  the  opera¬ 
tional  test  and  evaluation  required  under  subsection  (a). 

“(3)  The  costs  of  all  tests  required  under  that  subsec¬ 
tion  shall  be  paid  from  funds  available  for  the  system 
being  tested. 

“(?)  Waiver  Authority — The  Secretary  of  Defense  may 
waive  the  application  of  the  survivability  and  lethality  tests 
of  this  section  to  a  covered  system,  munitions  program,  or 
missile  program  if  the  Secretary,  before  the  system  enters 
full-scale  engineering  development,  certifies  to  Congress 
that  live-fire  testing  of  such  system  or  program  would  be 
unreasonably  expensive  and  impractical. 

“(d)  Waiver  in  Tune  of  War  or  Mobilization — In  time 
of  war  or  mobilization,  the  President  may  suspend  the 
operation  of  any  provision  of  this  section. 

“(e)  Definitions — In  this  secdon: 

“(1)  The  term  ‘covered  system*  means  a  vehicle, 
weapon  platform,  or  conventional  weapon  system — 

“(A)  that  includes  features  designed  to  provide 
some  degree  of  protection  to  users  in  combat;  and 


Preceding  page  blank 


67 


I 


68 

“(B)  that  is  a  major  system  within  the  meaning 
of  that  term  in  section  2303(5)  of  this  title 

“(2)  The  term  ‘major  munitions  program’  means — 
“(A)  a  munition  program  for  which  more  than 
1,000,000  rounds  are  planned  to  be  acquired;  or 

“(B)  a  conventional  munitions  program  that  is 
a  major  system  within  the  meaning  of  that  term  in  section 
2302(5)  of  this  title. 

“(3)  The  term  ‘major  defense  acquisition  program’ 
means — 

“(A)  a  conventional  weapons  system  that  is  a 
major  system  within  the  meaning  of  that  term  in  section 
2302(5)  of  this  title;  and 

“(B)  is  designed  for  use  in  combat. 

“(4)  The  term  ‘realistic  survivability  testing’  means, 
in  the  case  of  a  covered  system,  testing  for  vulnerability 
and  survivability  of  the  system  in  combat  by  firing  muni¬ 
tions  likely  to  be  encountered  in  combat  (or  munitions 
with  a  capability  similar  to  such  munitions)  at  the  system, 
configured  for  combat,  with  the  primary  emphasis  on 
testing  vulnerability  with  respect  to  potential  user  casual¬ 
ties  and  taking  into  equal  consideration  the  operational 
requirements  and  combat  performance  of  the  system. 

“(5)  The  term  ‘realistic  lethality  testing’  means,  in  the 
case  of  a  major  munitions  program  or  a  missile  program, 
testing  for  lethality  by  firing  the  munition  or  missile  con¬ 
cerned  at  appropriate  targets  configured  for  combat. 

“(6)  The  term  ‘configured  for  combat,’  with  respect 
to  a  weapon  system,  platform,  or  vehicle,  means  loaded 
or  equipped  with  all  dangerous  materials  (including  all 
flammables  and  explosives)  that  would  normally  be  on 
board  in  combat. 

“(7)  The  term  ‘operational  test  and  evaluation’  has 
the  meaning  given  that  term  in  section  138(a)(2)(A)  of 
this  title.” 

— (2)  The  table  of  sections  at  the  beginning  of  such  chapter 
is  amended  by  adding  after  the  item  relating  to  section  2365 
(as  added  by  section  909)  the  following  new  item: 

“2366.  Major  systems  and  munitions  programs:  surviv¬ 
ability  and  lethality  testing;  operational  testing.” 

(b)  Effective  Date — Section  2366  of  title  10,  United 
States  Code  (as  added  by  subsection  (a)),  shall  apply  with 
respect  to  any  decision  to  proceed  with  a  program  beyond 
low-rate  initial  production  that  is  made — 

(1)  after  May  31, 1987,  in  the  case  of  a  decision  re¬ 
ferred  to  in  subsection  (a)(1)  or  (a)(2)  of  such  section;  or 

(2)  after  the  date  of  the  enactment  of  this  Act,  in 

the  case  of  a  decision  referred  to  in  subsection  (a)(3)  of 
such  section.  , 

(c)  Time  for  Submission  of  Annual  Report  of  Director 
(OT&E)— Subsection  (g)(1)  of  section  138  of  such  title 
[as  redesignated  by  section  101(a)  of  the  Goldwater-Nich- 
ols  Department  of  Defense  Reorganization  Act  of  1986 
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(Public  Law  99-433)]  is  amended  by  striking  out  “January 
15”  in  the  second  sentence  and  all  that  follows  through 
‘is  prepared’  and  inserting  in  lieu  thereof  “10  days  after 
the  transmission  of  the  budget  for  the  next  fiscal  year 
under  section  1105  of  title  31.” 

FY88-89  DoD  Authorization  Act 

SEC.  802  SURVIVABILITY  AND  LETHALITY 
TESTING  OF  MAJOR  SYSTEMS 
(a)  Inclusion  of  Significant  Product  Improvement  Pro¬ 
grams — 

The  following  material  is  section  2366  of  title  10,  United 
States  Code,  as  amended  by  this  section.  The  changes 
are  in  italic. 

2366.  Major  systems  and  munitions  programs:  surviv¬ 
ability  and  lethality  testing;  operational  testing 
“(a)  Requirements — 

“(/)  The  Secretary  of  Defense  shall  provide  that — 
“(A)  a  covered  system  may  not  proceed  beyond 
low-rate  initial  production  until  realistic  survivability  test¬ 
ing  of  the  system  is  completed  in  accordance  with  this 
section; 

“(B)  a  major  munition  program  or  a  missile 
program  may  not  proceed  beyond  low-rate  initial  produc¬ 
tion  until  realistic  lethality  testing  of  the  program  is  com¬ 
pleted  in  accordance  with  this  section;  and 

“(C)  a  major  defense  acquisition  program  may 
not  proceed  beyond  low-rate  initial  production  until  initial 
operational  test  and  evaluation  of  the  program  is  com¬ 
pleted  in  accordance  with  this  section. 

‘Y 2)  The  Secretary  of  Defense  shall  provide  that  a 
covered  product  improvement  program  may  not  proceed 
beyond  low-rate  initial  production  until — 

“(A)  in  the  case  of  a  product  improvement  to 
a  covered  system ,  realistic  survivability  testing  is  com¬ 
pleted  in  accordance  with  this  section ;  and 

“(B)  in  the  case  of  a  product  improvement  to  a 
major  munitions  program  or  a  missile  program ,  realistic 
lethality  testing  is  completed  in  accordance  with  this 
section 

“(b)  Test  Guidelines — 

“(1)  Survivability  and  lethality  tests  required  under 
subsection  (a)  shall  be  carried  out  sufficiently  early  in  the 
development  phase  of  the  system  or  program  ( including  a 
covered  product  improvement  program)  to  allow  any  de¬ 
sign  deficiency  demonstrated  by  the  testing  to  be  corrected 
in  the  design  of  the  system,  munition,  or  missile  (or  in  the 
product  modification  or  upgrade  to  the  system ,  munition , 
or  missile)  before  proceeding  beyond  low-rate  initial  pro¬ 
duction. 

“(2)  In  the  case  of  a  major  defense  acquisition  pro¬ 
gram,  no  person  employed  by  the  contractor  for  the  sys¬ 
tem  being  tested  may  be  involved  in  the  conduct  of  the 
operational  test  and  evaluation  required  under  subsection 
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(a).  The  limitation  in  the  preceding  sentence  does  not 
apply  to  the  extent  that  the  Secretary  of  Defense  plans 
for  persons  employed  by  that  contractor  to  be  involved 
in  the  operation ,  maintenance ,  and  support  to  the  system 
being  tested  when  the  system  is  deployed  in  combat. 

“(3)  The  costs  of  all  tests  required  under  that  subsec¬ 
tion  shall  be  paid  from  funds  available  for  the  system 
being  tested. 

“(c)  Waiver  Authority — 

‘Y 1 )  The  Secretary  of  Defense  may  waive  the  applica¬ 
tion  of  the  survivability  and  lethality  tests  of  this  section 
to  a  covered  system,  munitions  program,  missile  program , 
or  covered  product  improvement  program  if  the  Secre¬ 
tary,  before  the  system  enters  full-scale  engineering  de¬ 
velopment,  certifies  to  Congress  that  live-fire  testing  of 
such  system  or  program  would  be  unreasonably  expensive 
and  impractical.  The  Secretary  shall  include  with  any 
such  certification  a  report  explaining  how  the  Secretary 
plans  to  evaluate  the  survivability  or  the  lethality  of  the 
system  or  program  and  assessing  possible  alternatives 
to  realistic  survivability  testing  of  the  system  or  program. 

“(2)  In  time  of  war  or  mobilization,  the  President  may 
suspend  the  operation  of  any  provision  of  this  section. 
‘Y  d)  Reporting  to  Congress — At  the  conclusion  of  surviv¬ 
ability  or  lethality  testing  under  subsection  (a),  the  Secre¬ 
tary  of  Defense  shall  submit  a  report  on  the  testing  to 
the  defense  committees  of  congress  (as  defined  in  section 
2362(e)(3)  of  this  title). 

“(e)  Definitions — In  this  section: 

“(1)  The  term  ‘covered  system’  means  a  vehicle, 
weapon  platform,  or  conventional  weapon  system — 

“(A)  that  includes  features  designed  to  provide 
some  degree  of  protection  to  users  in  combat;  and 

“(B)  that  is  a  major  system  within  the  meaning 
of  that  term  in  section  2303(5)  of  this  title 

“(2)  The  term  ‘major  munitions  program’  means — 
“(A)  a  munition  program  for  which  more  than 
1,000,000  rounds  are  planned  to  be  acquired;  or 

“(B)  a  conventional  munitions  program  that  is 
a  major  system  within  the  meaning  of  that  term  in  section 
2302(5)  of  this  title. 

“(3)  The  term  ‘major  defense  acquisition  program’ 
means — 

“(A)  a  conventional  weapons  system  that  is  a 
major  system  within  the  meaning  of  that  term  in  section 
2302(5)  of  this  title;  and 

“(B)  is  designed  for  use  in  combat 

“(4)  The  term  ‘realistic  survivability  testing’  means, 
in  the  case  of  a  covered  system  (or  a  covered  product 
improvement  program  fqra  covered  system ),  testing  for 
vulnerability  of  the  system  in  combat  by  firing  munitions 
likely  to  be  encountered  in  combat  (or  munitions  with 
a  capability  similar  to  such  munitions)  at  the  system. 


configured  for  combat,  with  the  primary  emphasis  on 
testing  vulnerability  with  respect  to  potential  user  casual¬ 
ties  and  taking  into  equal  consideration  the  susceptibility 
to  attack  and  combat  performance  of  the  system. 

“(5)  The  term  ‘realistic  lethality  testing’  means,  in 
the  case  of  a  major  munitions  program  or  a  missile  pro¬ 
gram  (or  a  covered  product  improvement  program  for 
such  a  program ),  testing  for  lethality  by  firing  the  muni¬ 
tion  or  missile  concerned  at  appropriate  targets  configured 
for  combat. 

“(6)  The  term  ‘configured  for  combat,’  with  respect 
to  a  weapon  system,  platform,  or  vehicle,  means  loaded 
or  equipped  with  all  dangerous  materials  (including  all 
flammables  and  explosives)  that  would  normally  be  on 
board  in  combat. 

“(7)  The  term  ‘operational  test  and  evaluation’  has 
the  meaning  given  that  term  in  section  138(a)(2)(A)  of 
this  title.” 

“(8)  The  term  * covered  product  improvement  pro¬ 
gram *  means  a  program  under  which — 

“(A)  a  modification  or  upgrade  will  be  made 
to  a  covered  system  which  (as  determined  by  the  Secretary 
of  Defense)  is  likely  to  affect  significantly  the  survivability 
of  such  system ;  or 

“(B)  a  modification  or  upgrade  will  be  made 
to  a  major  munitions  program  ora  missile  program  which 
(as  determined  by  the  Secretary  of  Defense)  is  likely  to 
affect  significantly  the  lethality  of  the  munition  or  missile 
produced  under  the  program. 

— (2)  The  table  of  sections  at  the  beginning  of  such 
chapter  is  amended  by  adding  after  the  item  relating  to 
section  2365  (as  added  by  section  909)  the  following  new 
item: 

“2366.  Major  systems  and  munitions  programs:  surviv¬ 
ability  and  lethality  testing;  operational  testing.” 

(b) Effective  Date — Section  2366  of  title  10,  United  States 
Code  (as  added  by  subsection  (a)),  shall  apply  with  re¬ 
spect  to  any  decision  to  proceed  with  a  program  beyond 
low-rate  initial  production  that  is  made — 

(1)  after  May  31, 1987,  in  the  case  of  a  decision  re¬ 
ferred  to  in  subsection  (a)(1)  or  (a)(2)  of  such  section;  or 

(2)  after  the  date  of  the  enactment  of  this  Act,  in 
the  case  of  a  decision  referred  to  in  subsection  (a)(3)  of 
such  section. 

(c)  Time  for  Submission  of  Annual  Report  of  Director 
(OT&E) — Subsection  (g)(1)  of  section  138  of  such  title 
[as  redesignated  by  section  101(a)  of  the  Goldwater-Nich- 
ols  Department  of  Defense  Reorganization  Act  of  1986 
(Public  Law  99-433)]  is  amended  by  striking  out  “January 
15”  in  the  second  sentence  and  all  that  follows  through 
‘is  prepared’  and  inserting  in  lieu  thereof  “10  days  after 
the  transmission  of  the  budget  for  the  next  fiscal  year 
under  section  1105  of  title  31  ” 


The  General  Accounting  Office  Study: 
Live  Fire  Testing; 

Evaluating  DoD’S  Programs 


The  General  Accounting  Office  (GAO)  study  was  conducted  at  the  request  of  the  Chair¬ 
man,  Seapower  Subcommittee,  House  Armed  Services  Committee.  The  purpose  of  this  study 
was  to  answer  four  questions:  (1)  What  is  the  status  of  each  system  originally  scheduled  for 
live  fire  testing  under  the  JLF  program?  (2)  What  has  been  the  methodological  quality  of  the 
test  and  evaluation  process?  (3)  What  are  the  advantages  and  limitations  of  full-up  live  fire 
testing,  and  how  do  other  methods  complement  full-up  testing?  (4)  How  can  live  fire  testing 
be  improved?  Of  interest  here  are  questions  2,  3,  and  4.  The  conclusions  presented  by  GAO 
for  each  of  these  three  questions  for  the  aircraft  portion  of  its  study  are  given  below.  The  abbre¬ 
viations  DTPS  and  V/L  stand  for  detailed  test  plans  and  vulnerability/lethality,  respectively. 


What  Has  Been  the  Methodological  Quality  of  the  Test  and  Evaluation  Process? 

AIRCRAFT  SPECIFIC 

Overall  Planning 

•  In  general,  JLF/Aircraft  planning  has  been  well  orga¬ 
nized  and  thorough. 

•  JLF/Aircraft  established  a  formal  process  to  desig¬ 
nate  test  priorities;  however,  test  priorities  were  actu¬ 
ally  driven  by  more  pragmatic  concerns  (target  avail¬ 
ability  and  the  need  to  ensure  tri-service  cooperation). 

•  The  principal  constraint  on  realism  is  the  inability 
to  simulate  flight  conditions  on  the  ground.  Airflow  is 
used  to  simulate  airspeed  but  the  coverage  area  is  small, 
and  other  environmental  factors  affecting  Are  are  not 
simulated  at  all. 

Setting  Test  Objectives 

•  In  FY85  and  FY86  DTPS,  JLF/Aircraft  specified 
objectives  congruent  with  the  version  of  the  program 
objectives  they  had  established.  These  were  generally 
feasible,  with  the  exception  of  objectives  related  to 
determining  probabilities. 
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Test  Planning  •  JLF/Aircraft  test  designs  are  generally  congruent 

with  test  objectives,  efficient  with  respect  to  conserving 
targets,  and  realistic  given  their  limited  objectives. 

•  Some  DTPS  specified  target  requirements  which 
exceeded  the  availability  of  those  targets. 

•  Testers  are  highly  sensitive  to  test  efficiency  from  an 
engineering  standpoint,  i.e.,  designing  tests  to  conserve 
targets  and  prevent  testing  effects. 

•  DTPS  omit  key  information  (e.g..  data  analysis  plans) 
and  are  inconsistent  in  selection  of  threat  velocities. 

Implementation  •  To  the  limited  extent  we  could  observe  them,  depar¬ 

tures  from  test  plans  have  generally  been  reasonable. 


Analysis  and  Results 

•  Only  one  draft  report  has  been  completed  —  the 

F 100  engine  steady  state  fuel  ingestion  tesL  This  report 
omitted  key  information,  overstated  the  generaiizabil- 
ity  of  results,  and  presented  a  highly  questionable 
mode.  Recommendations  were  congruent  with  results 
and  sensitive  to  the  likelihood  of  user  acceptance. 

GENERAL  LFT  ISSUES 

Conflict  over  Objectives 

•  The  JLF  charter  did  not  define  live  fire  testing  well 
enough  to  give  test  designers  a  clear  direction. 

•  There  have  been  several  conflicting  versions  of  the 
objectives  of  JLF  and  live  fire  testing  in  general.  This 
appears  to  have  in  part  resulted  from  the  decision  to 
task  the  JTCG’s  to  implement  JLF. 

•  The  conflict  over  objectives  reflects  underlying  dif¬ 
ferences  between  the  interests  of  proponents  of  full- 
up  testing  and  those  of  modelers,  resulting  in  largely 
incompatible  approaches. 

Availability  of  Targets 

•  The  principal  constraint  faced  by  all  JLF  test  officials 
is  a  lack  of  targets.  This  is  in  part  a  result  of  inadequate 
planning;  there  is  no  assigned  responsibility  to  provide 
targets  and  related  support  to  JLF.  Consequently,  test 
officials  have  had  to  spend  a  substantial  portion  of 
their  time  “selling”  the  program  to  skeptical  service 
components. 

•  The  systems  and  components  that  JLF  does  receive 
are  frequently  in  poor  condition,  yet  JLF  provided  no 
funds  for  restoration. 

•  JLF  has  been  further  hindered  by  competing  govern¬ 
mental  and  non-govemmental  interests  and  negative 
attitudes  toward  destructive  testing. 

Statistical  Validity 

•  In  general,  the  sample  sizes  of  JLF  and  related  live 
fire  testing  have  not  been  sufficient  to  produce  statisti¬ 
cally  reliable  results.  This  would  be  a  problem  even 
if  the  number  of  targets  listed  in  the  test  plans  could 
be  obtained. 

•  The  statistical  input  to  JLF  has  been  minimal  and 
had  little  effect,  and  the  few  applications  of  statistical 
analysis  to  live  fire  test  data  thus  far  are  highly  ques¬ 
tionable.  Several  efforts  are  underway  to  make  live 
fire  tests  more  statistically  interpretable. 

•  As  a  substitute  for  statistical  analysis,  engineering 
judgment —  which  is  heavily  relied  upon  through  the 
V/L  process  —  has  little  scientific  validity,  being  sub¬ 
ject  to  individual  and  collective  biases. 

•  The  most  common  form  of  vulnerability /lethality 
indicator— probability  of  kill  given  a  hit  P**  has  not 
been  demonstrated  to  be  reliable  or  valid. 
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Shot  Selection 

•  Controversy  over  shot  selection  is  to  some  degree 
a  conflict  between  sampling  efficiency  and  the  desire 
to  avoid  bias  at  all  costs. 

•  Random  sampling  from  combat  distributions  is  a 
reasonable  way  to  preclude  intentional  or  inadvertent 
bias  in  shot  selection.  However,  sampling  from  a  uni* 
form  distribution  avoids  tester  bias  and  biases  in  the 
combat  data. 

•  The  shot  selection  problem  will  not  be  resolved  by 
technical  solutions  alone.  .An  interim  solution  might  be 
to  designate  that  some  proportion  of  shots  be  selected 
judgmentaily  and  others  randomly,  but  ultimately,  it 
appears  impossible  to  agree  on  how  to  select  live  fire 
shots  without  first  deciding  on  test  objectives. 

Human  Effects 

•  JLF  plans  do  not  provide  an  adequate  treatment  of 
human  effects. 

•  The  claims  of  some  JLF  officials  that  personnel 
vulnerability  is  well  known  are  overstated. 

•  Given  the  current  state  of  the  art,  it  is  unlikely  that 
JLF  will  produce  precise  estimates  of  casualties. 

Incentive  Structure 

•  DoD’s  incentive  structure  is  not  entirely  conducive 
to  realistic  live  fire  testing. 

COMPARISON  PROGRAMS 

Past  Programs 

•  The  state  of  the  art  of  live  fire  testing  has  improved 
since  prior  live  fire  testing  programs,  but  some  poten¬ 
tially  solvable  problems  raised  earlier  have  not  been 
solved.  For  example,  little  progress  has  been  made  in 
the  empirical  validation  of  V/L  estimates  (P^s) 

SUMMARY  CONCLUSION 

•  There  is  little  completed  testing  on  which  to  base  a 
methodology  evaluation.  However,  it  is  apparent  that 
the  technical  capability  to  do  full-up  testing  is  not  well 
developed.  This  is  partly  due  to  the  historically  low 
emphasis  on  live  fire  testing  in  the  U.S. 
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What  Are  the  Advantages  and  Limitations  of  Full-up  Live  Fire  Testing? 

(Both  Land  and  Air  Targets) 

ADVANTAGES  OF  LrVE  FIRE  TESTING 

•  As  the  only  method  providing  direct  visual  observa¬ 
tion  of  the  damage  caused  by  a  weapon/target  interac¬ 
tion  under  realistic  combat  conditions,  full-up  live  fire 
testing  offers  a  unique  advantage  overall  other  methods 
of  V/L  assessment. 

•  The  descriptions  of  directly  observable  damage  that 
fuil-up  testing  provides  are  regarded  as  highly  benefi¬ 
cial  by  users. 

•  Full-up  testing  has  already  demonstrated  some  value 
by  producing  several  '‘surprises,'’  i.e..  results  that  were 
not  predicted,  and  might  not  have  been  detected  by 
other  methods  of  testing  or  analysis. 

LIMITATIONS  OF  LIVE  FIRE  TESTING 

High  Cost 

•  The  primary  limitation  of  full-up,  full-scale  live  fire 
testing  is  cost.  On  a  per  shot  basis,  it  is  considerably 
more  expensive  than  inert  or  subscale  testing,  primarily 
due  to  the  high  cost  and  limited  availability  of  targets. 
Testing  and  restoration  costs  are  also  higher,  as  are 
their  associated  time  requirements.  Nonetheless,  live 
fire  testing  costs  are  a  very  small  percentage  of  total 
program  costs. 

Limited  Information 

•  Full-up  testing  potentially  yields  less  information 
about  damage  mechanisms  per  shot  than  inert  or  sub¬ 
scale  testing,  primarily  because  catastrophic  kills  de¬ 
stroy  the  target  and  its  components,  along  with  much 
of  the  instrumentation  used  to  record  the  damage. 
However,  not  all  full- up  shots  result  in  catastrophic 
kills;  such  shots  potentially  yield  more  interpretable 
information  than  equivalent  inert  shots. 

Limited  Generaiizability 

•  Full-up  live  fire  test  results  typically  are  less  easily 
generalized  beyond  the  specific  test  conditions  than 
inert  or  subscale  testing.  Full-up  testing  brings  a  larger 
number  of  variables  into  play  that  potentially  affect 
outcomes,  yet  because  full-up  testing  destroys  targets, 
a  smaller  proportion  of  relevant  test  conditions  can  be 
examined. 

Limited  Redesign  Opportunities 

•  The  impact  of  live  fire  testing  of  developed  systems 
is  limited  by  “frozen’*  designs  which  are  prohibitively 
expensive  to  change.  For  this  reason,  test  officials  see 
the  main  benefit  of  JLF  and  related  programs  as  reduc¬ 
ing  vulnerability  of  future  systems  through  lessons 
learned.  This  is  not  to  suggest,  however,  that  important 

V/L  modifications  are  never  feasible. 
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How  Do  Other  Methods  Complement  Full-up  Testing? 

(Both  Land  and  Air  Targets) 

Subscale  Testing 

•  Subscaie  tests  can  support  larger  sample  sizes  than 
full-scale  tests  (whether  fuli-up  or  inert),  and  are  useful 
tn  bounding  effects  and  providing  input  to  models. 
Certain  types  of  subscale  testing  are  also  useful  for 
developing  generic  characterization  of  munitions  ef¬ 
fects. 

•  Subscale  tests  can  provide  only  indirect  evidence  of 
synergistic  effects  on  realistic  targets,  which  must  be 
inferred  through  an  unproven  analytical  process  (mod¬ 
eling).  Therefore,  subscale  testing  can  supplement 
full-up.  full-scale  testing  but  not  substitute  for  it. 

Inert  Testing 

•  Inert  testing  of  full-scale  targets  is  superior  to  full- 
up  testing  in  characterizing  mechanical  damage  to  indi¬ 
vidual  components  and  in  conserving  both  components 
and  targets. 

•  Catastrophic  damage  cannot  be  observed  directly 
from  shots  on  inert  targets,  and  the  standard  method 
for  inferring  a  K-kill  underestimates  its  true  likelihood. 
Like  subscale  tests,  inert  tests  can  provide  only  indirect 
evidence  of  effects  on  realistic  (i.e..  full-up)  targets, 
inferred  through  models  acknowledged  to  be  weak  on 
combustibles.  Therefore,  inert  testing  can  supplement 
full-up,  full-scale  testing  but  not  substitute  for  it 

Combat  Data 

•  Analysis  of  combat  data,  if  available,  has  several 
advantages  over  V/L  testing:  it  provides  greater  real¬ 
ism,  includes  information  above  the  level  of  vulnerabil¬ 
ity  and  lethality  (e.g.,  aggregated  survivability  mea¬ 
sures).  and  is  considerably  less  expensive. 

•  Combat  data  provide  less  scientific  control  than  test¬ 
ing.  are  limited  to  munitions  and  systems  that  have 
been  employed  in  combat,  and  offer  no  direct  view  of 
the  damage  process  or  the  conditions  of  firing.  Like 
subscale  and  inert  testing,  combat  data  can  supplement 
full-up.  full-scale  testing  but  not  substitute  for  it 

Modeling 

•  V/L  models  support  the  design  and  interpretation  of 
live  fire  tests,  and  are  potentially  useful  in  extrapolating 
beyond  test  results.  A  unique  advantage  of  models  over 
testing  is  their  applicability  to  systems  not  yet  built 

•  Models  are  widely  used  in  V/L  assessment  generally, 
but  play  a  more  central  role  in  the  design  and  interpreta¬ 
tion  of  armor  tests  than  in  aircraft  tests. 

•  It  does  not  appear  that  models  have  as  yet  played 
as  great  a  role  in  the  design  of  live  fire  tests  as  some 
statements  by  the  modelers  would  indicate. 

•  Current  vulnerability  models  share  numerous  limita¬ 
tions;  specifically,  fire,  explosion,  multiple  hits,  rico¬ 
chets,  synergistic  effects,  and  human  effects  are  not 
yet  well  modeled. 

•  Many  of  the  most  important  mechanisms  for  pro¬ 
ducing  casualties  are  poorly  modeled,  if  at  all.  Without 
specific  efforts  to  bring  these  casualty  mechanisms  into 
the  modeling  process,  V/L  models  can  be  expected  to 
be  of  limited  utility  in  predicting  casualty  reduction. 

•  Currently  used  V/L  models  are  inadequately  vali¬ 
dated. 
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•  A  large  part  of  the  modeling  and  model  revision 
process  is  closed  to  outside  analysis,  including  weapon 
designers.  This  has  led  to  claims  that  modelers  ignore 
or  mis  specify  important  V/L  mechanisms,  or  that  they 
are  accountable  only  to  their  own  community. 

•  Claims  that  vulnerability  models  predict  poorly  are 
somewhat  overstated,  often  referring  to  predictions 
from  older  models  not  expected  to  be  used  in  live  fire 
tests,  and  insufficient  test  or  combat  data  to  permit 
unqualified  conclusions.  Additionally,  little  attention 
has  been  paid  to  the  different  levels  of  accuracy  re¬ 
quired  for  different  user’s  purposes. 

•  The  stochastic  components  introduced  into  vulnera¬ 
bility  models  after  the  Bradley  Phase  I  tests  provide 
an  unknown  level  of  protection  from  invalidation  by 
test  data. 

•  There  are  no  clearly  specified  mechanisms  for  using 
live  fire  test  data  to  calibrate  or  revise  models. 
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How  Can  Live  Fire  Testing  Be  Improved? 

TECHNICAL  IMPROVEMENTS 

We  suggest  that  DoD 

..  • 

1 .  Improve  the  estimation  of  human  effects.  Begin  by 
replacing  nomnstnimented  plywood  mannequins  with 
the  instrumented  anthropomorphic  type. 

2.  Improve  the  reliability  and  validity  of  quantitative  V/ 

L  estimates.  For  example,  interrater  agreement  studies 
could  determine  the  magnitude  of  the  reliability  prob¬ 
lem.  and  provide  insights  into  reducing  it 

3.  Expand  efforts  to  improve  statistical  validity,  and 
establish  guidelines  for  the  statistical  interpretation  of 
small-sample  live  fire  test  results. 

4.  Concentrate  model  improvements  on  currently  weak 
areas  vital  to  casualty  estimation — fire  and  explosion 
and  human  effects. 

5.  Establish  guidelines  for  how  models  can  better  sup¬ 
port  the  design  and  interpretation  of  live  fire  tests. 

6.  Establish  guidelines  for  how  live  fire  test  results  can 
be  used  in  the  revision  of  models. 

7.  Allow  outside  analysis  into  the  modeling  and  model¬ 
ing  revision  process,  and  provide  better  documentation 
of  the  process  for  use  by  those  analysts. 

8.  Accumulate  comparisons  of  model  predictions  with 
live  fire  test  results  over  multiple  tests  in  order  to  assess 
improvements  in  models,  and  make  results  available 
to  outside  analysts:  also  redo  predictions  of  earlier  live 
fire  shots  after  models  have  been  revised  in  order  to 
validate  improvements. 

9.  Require  that  detailed  test  plans  include  shotlines, 
munitions,  sample  sizes,  predictions,  analysis  plans, 
rationales  for  decisions,  and  other  critical  information 
to  enable  proper  oversight  Keeping  plans  unclassified 
should  not  be  a  justification  for  omitting  key  infor¬ 
mation. 

10.  Develop,  modify,  or  procure  instrumentation  to 
yield  more  information  from  catastrophic  shots. 

11.  Improve  methods  for  simulating  in-flight  condi¬ 
tions;  specifically  altitude,  altitude  history,  maneuver 
load,  and  slosh. 

GENERAL  IMPROVEMENTS 

We  suggest  that  DoD 

1.  Avoid  requiring  unrealistic  or  incompatible  objec¬ 
tives  in  future  live  fire  tests  (e.g.,  combat  realism  and 
model  validation). 

2.  Consider  total  program  costs  in  considerations  of 
target  costs,  including  for  example  the  concept  of  a 
percentage  set-aside  for  live  fire  testing. 

3.  Determine  whether  the  live  fire  testing  infrastructure 
is  adequate  to  implement  the  legislation,  or  has  to  be 
expanded.  For  example,  only  two  facilities  in  the  U.S. 
currently  have  high  speed  airflow  capability. 

4.  Determine  to  the  extent  possible  the  cost  of  Live 
fire  testing  of  new  systems,  and  the  relative  costs  and 
benefits  of  different  approaches  to  live  fire  testing. 
Currently,  there  are  claims  and  counter-claims  about 
the  costs  of  full-up  vs.  subscale  tests,  but  little  data. 

5.  Promote  awareness  of  the  benefits  to  be  obtained 
from  destructive  testing  to  top  level  military  and  civil¬ 
ian  officials. 

6.  With  the  legislation  as  a  foundation,  continue  to 
strengthen  incentives  that  support  realistic  live  fire 
testing. 
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Recommendations  to  the  Secretary  of  Defense 
[n  addition  to  the  improvements  noted  elsewhere,  there 


is  a  need  to  resolve  current  conflicts  about  the  purpose 
of  live  fire  tests  and  to  make  clear  that  the  objective 
of  reducing  vulnerability  and  increasing  lethality  of 
U.S.  systems  is  the  primary  emphasis  of  testing.  Ac¬ 
cordingly,  we  recommend  that  the  Secretary  of  Defense 


1.  Conduct  full- up  tests  of  developing  systems,  first  at 
the  subscale  level  as  subscale  systems  are  developed, 
and  later  at  the  full-scale  level  mandated  in  the  legisla¬ 
tion.  This  will  minimize  vulnerability  “surprises”  at 
the  full-scale  level,  at  which  time  design  changes  are 
more  difficult  and  costly. 

2.  Establish  guidelines  on  the  role  live  fire  testing  will 
play  in  procurement. 

3.  Establish  guidelines  on  the  objectives  and  conduct 
of  live  fire  testing  of  new  systems,  with  particular 
attention  to  clarifying  what  is  to  be  expected  from  the 
services. 

4.  Ensure  that  the  primary  users*  priorities  drive  the 
objectives  of  live  fire  tests.  Modelers  are  secondary 
users. 

5.  Recent  live  fire  legislation  requires  the  services  to 
provide  targets  for  testing  new  systems,  but  there  is 
no  similar  requirement  for  the  fielded  systems  in  JLF, 
where  lack  of  targets  has  impeded  testing.  Accordingly, 
we  recommend  that  the  Secretary  of  Defense  provide 
more  support  to  JLF  for  obtaining  targets. 


References 
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Board  on  Army  Science  and  Technology 
Review  of  the  Army’s  Assessment 
Methodology  for  Combat  Vehicle 
Vulnerability  to  Anti- Armor  Weapons 


“During  the  combat-loaded,  live-fire  testing  of  the  Bradley  Fighting  Vehicle  System 
(BFVs)  in  1986,  differences  of  viewpoint  arose  among  the  Office  of  the  Secretary  of  the 
Army,  the  Office  of  the  Secretary  of  Defense,  and  the  Congress  regarding  the  role  and  conduct 
of  live-ammunition  firings  and  the  interpretation  of  results  in  assessing  the  vulnerability  of 
armored  vehicles.  To  resolve  these  differences,  the  National  Research  Council’s  Board  on 
Army  Science  and  Technology  was  requested  by  Walter  W.  Hollis,  Deputy  Under  Secretary 
of  the  Army  (Operations  Research),  to  examine  and  make  recommendations  concerning 
the  Army’s  assessment  of  vulnerability  of  armored  vehicles  against  anti-armor  weapons. 
Accordingly,  a  Committee  on  the  Review  of  Army  Vulnerability  Assessment  Methods  was 
formed  to  conduct  the  necessary  studies"  (NRC,  1989).  The  committee  was  tasked  to  conduct 
a  review  independently  of  the  Army’s  in-house  laboratories  and  contractors  to  (1)  address 
issues  that  will  help  the  Army  define  the  objectives  of  its  vulnerability  assessment  program, 
(2)  define  and  analyze  alternative  ways  to  balance  computation  and  live  fire  testing  in  reaching 
conclusions  about  vehicle  vulnerability,  (3)  identify  technical  deficiencies  where  they  exist, 
and  (4)  suggest  alternatives  for  improvement  as  appropriate.  The  conclusions  reached  by  the 
committee  are  given  below  (NRC,  1989). 


1. 

A  dear  distinction  must  be  made  between  (a)  live  firings  conducted  against  fully  combat-loaded 
vehicles,  and  (b)  live  firings  conducted  against  vehicle  components,  subsystems,  and  prototypes 
during  the  course  of  engineering  design  and  development.  The  latter  tests  are  intended  to  provide 
engineering  information  at  minimum  cost  and  expenditure  of  resources.  Combat-loaded.  Live-fire 
tests  are  for  a  different  purpose,  namely,  to  provide  an  independent  check  on  the  general  success 
of  the  design  and  development  process  with  regard  to  the  vulnerability  of  the  vehicle  to  enemy 
fire  with  threat  weapons  likely  to  be  encountered  on  the  battlefield. 

2. 

The  preparation  of  a  ROC  (Required  Operational  Capability)  precedes  the  initiation  of  design 
and  development  of  a  new  combat  vehicle.  All  too  often,  the  ROC  underestimates  the  importance 
of  emerging  armor/anti-armor  technologies.  The  result  is  that  by  the  time  the  vehicle  is  ready  to 
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be  fielded,  the  threat  environment  is  more  severe  and  perhaps  even  different  in  nature  than  that 
called  for  in  the  ROC.  It  is  recommended  that,  as  pan  of  the  design  process,  future  designs  allow 
for  enhancement  of  ballistic  protection  during  the  vehicle’s  lifetime. 

3.  It  is  important  that  vulnerability  assessments  of  combat  vehicles  be  as  dependable  as  possible. 

Not  only  are  they  important  in  defining  the  hazards  to  crew  and  vehicle,  they  are  essential  in 
assessing  vehicle  survivability  on  the  battlefield,  as  well  as  for  many  other  important  Army 
planning  purposes.  A  complete  description  of  the  vulnerability  of  a  vehicle  is.  however,  a  complex 
task.  For  each  attacking  weapon  and  warhead  combination,  the  damage  due  to  attack  from  all 
directions  and  at  all  ranges  must  be  taken  into  account.  To  arrive  at  vulnerability  assessments, 
therefore,  the  only  recourse  is  to  make  use  of  mathematical  models  capable  of  being  executed 
on  a  high-speed  computer,  so  that  the  damage  due  to  large  numbers  of  attacks  can  be  assembled. 
Such  models  must  be  supported  by  an  adequate  base  of  data  obtained  experimentally  by  firings 
against  armor  samples,  components,  and  subsystems. 


Vulnerability  models  designed  to  at  least  two  levels  of  comprehensiveness  are  required.  For 
preliminary  design  purposes,  a  model  is  needed  which  provides  relatively  rapid  estimates  at  an 
accuracy  level  sufficient  to  compare  the  relative  advantages  of  competing  concepts.  At  the  opposite 
end  of  the  spectrum,  a  more  detailed  model  is  needed  for  assessing  the  vulnerability  of  a  design 
with  best  achievable  accuracy.  The  committee  has  concluded  that  suitable  models  for  addressing 
these  needs  are  not  currently  available  and  further  development  is  needed. 


The  committee  has  reviewed  the  current  BRL  approach  to  more  accurate  model  building.  It  is, 
in  essence,  based  on  the  belief  that  better  accuracy  will  result  from  models  of  increasing  detail, 
i.e.,  models  that  incorporate  the  vehicle  exterior  and  interior  geometry  in  relatively  minute  detail 
and  that  trace  behind-armor  damage  virtually  fragment  by  fragment  It  is  the  committee's  opinion 
that  such  an  approach  is  not  justified  because  of  the  inability  to  forecast  with  precision  the 
characteristics  and  performance  of  ever-evolving  threat  weapons,  and  because  of  the  inherently 
stochastic  nature  of  penetration  and  behind-armor  damage  mechanisms.  The  trend  toward  increas¬ 
ingly  detailed  models  is  not  a  productive  direction  and  the  committee  suggests  that  BRL  reconsider 
its  current  direction  for  model  design.  A  lesser  degree  of  detail,  using  an  approach  based  on  a 
more  generic  assessment  of  the  vulnerability  of  major  components,  would  still  provide  valid 
vulnerability  estimates  with  reduced  data  requirements  and  shorter  computational  times. 


BRL  is  the  Army's  principal  laboratory  responsible  for  armor/anti-armor  technology.  Based  on 
a  review  of  the  BRL  data,  encompassing  unclassified  as  well  as  classified  data,  but  not  including 
sensitive  compartmented  information,  it  is  evident  that  there  is  a  significant  lack  of  experimental 
information,  particularly  concerning  the  more  sophisticated  armor  designs  and  anti-armor  weapons 
represented  representative  of  modem  practice.  A  principal  reason  appears  to  be  that  in  recent 
yean  the  experimental  work  has  tended  to  be  conducted  on  an  ad  hoc  basis  for  different  develop¬ 
ment  programs.  The  experimental  research  program  that  has  been  instituted  to  establish  an 
integrated  data  base  for  use  as  a  reference  source  for  future  designs  and  as  a  guide  for  formulating 
further  research  efforts  has  been  neither  coordinated  nor  comprehensive.  The  inadequacy  of  this 
experimental  program  is  the  largest  single  deficiency  contributing  to  uncertainty  in  our  current 
vulnerability  estimates. 


One  purpose  of  this  study  is  to  better  define  the  role  of  live-fire  tests  against  fully  combat-loaded 
prototype  vehicles.  It  is  important,  therefore,  to  carefully  delineate  what  functions  these  tests 
fulfill  and.  equally  important,  what  they  do  not  add  to  the  process  of  vulnerability  assessment 
Specifically,  combat-loaded,  live-fire  tests  do  not  contribute  significantly  to  the  assessment  of 
vulnerability  in  a  form  needed  to  support  subsequent  survivability  assessments  and  for  other 
necessary  Army  uses.  The  quantity  of  data  gathered  by  such  tests  is  too  limited  in  scope  and 
depth  to  be  statistically  significant 


Combat-loaded,  live-fire  tests  will  accomplish  the  following,  provided  the  test  series  consists  of 
randomly  selected  firings  with  shotlines  selected  by  the  procedure  outlined  in  Chapter  5  or  its 
equivalent: 

•  During  the  interval  between  the  start  of  the  development  and  design  process  and  the  live-fire 

tests,  the  threat  environment  on  the  battlefield  may  have  changed  appreciably.  Since  the  combat- 
loaded,  live-fire  tests  are  to  be  conducted  with  weapons  constituting  updated  threat  weapons, 
they  provide  some  assessment  of  the  vehicle  performance  with  regard  to  vulnerability  to  weapons 
not  incorporated  in  the  ROC  document  •  . 

•  The  tests  are  conducted  in  an  environment  of  high  visibility  within  the  Department  of  Defense. 
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Congress,  and.  save  for  the  limitations  of  classified  data,  the  public  at  large.  Knowing  that  the 
test  results  will  be  carefully  observed  during  the  approval  process  leading  to  large-scale  production, 
the  program  manager  and  his  staff  will  be  motivated  to  ensure  that  adequate  weight  is  given  to 
vulnerability  considerations  throughout  the  design  process. 

•  The  tests  may  uncover  vulnerabilities  that  have  not  been  anticipated  and  that  represent  design 
deficiencies.  Experience  to  date  has  in  fact  shown  that  valuable  information  of  this  kind  has 
emerged  from  combat- loaded  tests. 

•  The  results  of  combat- loaded,  live-fire  tests  should  not  by  themselves  be  construed  as  a  basis 
for  approval  or  disapproval  of  the  transition  to  full-scale  production.-  Many  additional  factors 
must  be  taken  into  account  in  arriving  at  this  decision. 


9. 

Combat-loaded,  live-fire  tests  do  not  provide  information  of  significant  value  for  validating 
vulnerability  models,  although  they  may  disclose  vulnerabilities  which  have  been  overlooked  in 
model  formulations.  The  committee  recommends  that  such  tests  should  not  be  conducted  with 
this  purpose  in  mind 

10. 

Experience  to  date  with  combat-loaded  live-fire  tests  has  indicated  that  they  do  produce  positive 
findings  helpful  in  reducing  vehicle  vulnerability.  Many  of  the  findings,  however,  could  have 
been  anticipated  by  more  careful  engineering  testing  conducted  earlier  and  with  substantially 
lower  expenditure  of  resources. 

11. 

To  improve  future  design  practices,  and  particularly  to  help  less  experienced  designers  without 
extensive  “corporate"  experience,  the  committee  recommends  preparation  of  a  manual  of  good 
design  practices  for  combat  vehicles  to  reduce  the  vulnerability  to  penetrating  rounds.  Reflecting 
a  compilation  of  sound  design  rules,  as  well  as  practices  to  be  avoided,  such  a  manual  will  help 
to  prevent  future  mistakes  that  might  result  in  increased  vulnerability. 
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Preliminary  Recommendations  from  the 
JTCG/AS  Pyh  Workshop 


Preliminary  Recommendations  from  the  Fuel  System 
Working  Panel  of  the  1991  JTCG/AS  P**  Workshop: 

•  Develop  the  Ullage  Explosion  Model  to  calculate  ul¬ 
lage  vulnerability  to  explosions  from  API,  HE,  and  fragment 
impacts. 

•  Reexamine  the  BRL  Void  Space  Fire  data  to  determine 
that  the  probabilities  for  fire  due  to  fragments  and  23-milli- 
meter  HE  projectiles  are  consistent  with  other  data. 

•  Modify  the  CO V ART  model  to  handle  the  simultane¬ 
ous  analysis  of  thermal  and  mechanical  damage  to  compo¬ 
nents. 

•  Determine  how  the  BRL  void  space  fire  probabilities 
need  to  be  modified  for  JP-5  and  JP-8  fuel. 

•  Determine  how  the  BRL  void  space  fire  probabilities 
will  change  from  open  to  closed  void  spaces. 

•  Determine  the  probability  of  fire  initiation  due  to 
“mixed  functioning”  of  the  incendiaries. 

•  Develop  a  methodology  to  determine  the  probability 
of  fires  due  to  secondary  ignition  sources. 

•  Develop  data  curves  for  the  vulnerability  of  fuel  sys¬ 
tem  hardware  components.* 

•  Determine  how  the  BRL  void  space  fire  probabilities 
will  change  if  the  “striker  plate”  consists  of  composite  mate¬ 
rials. 

•  Perform  a  test  program  to  determine  the  probability 


of  fire  initiation  due  to  a  “mixed  functioning”  of  the  incendi¬ 
aries. 

•  Determine  if  the  bum  characteristics  of  composite  ma¬ 
terials  are  sufficiently  different  to  warrant  a  change  in  the 
BRL  void  space  probability  numbers. 

•  Determine  how  the  altitude  will  affect  the  probability 
of  fire  initiation,  the  probability  that  fires  will  sustain,  and 
the  damage  that  might  result  from  sustained  fires. 

•  Determine  and  quantify  the  vulnerability  of  fuel  sys¬ 
tem  components  due  to  directed  energy  weapons. 

The  following  findings  of  the  eight  working  panels  are 
taken  from  a  preliminary  copy  of  a  briefing  being  prepared 
to  be  given  to  the  JTCG/AS  Central  Office  and  to  OSD. 
This  briefing  material  was  furnished  by  Gerald  Bennett, 
ASD/XRM,  WPAFB. 


Fuel  System  Panel 

•  Empirically  based  fuel/fire/explosion  Pw  data  exist 
that  are 

1.  more  than  15  years  old, 

2.  for  metallic  structures  only, 

3.  for  sea  level  only. 

•  No  traceable  fuel  system  hardware  Pw  data  exist. 
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Flight  Controls  and  Hydraulics  Panel 

•  Actuators 

1.  Analyses  are  only  on  older  (1950s)  designs. 

2.  Sparse  test  data  do  not  match  actuators  analyzed. 

3.  New  designs  and  technologies  have  no  analyses  or 
tests. 

•  Hydraulic  fluid  reservoirs,  and  accumulators 

1.  One  set  of  experimentally  validated  fluid  P^,  data 
exists. 

2.  Other  components  have  no  documented  test  data 
or  analyses. 

•  Cables,  bellcranks,  and  mechanical  components 

1.  Analytic  models  and  some  P^,  data  exist 

2.  Selective  tests  needed  to  validate  analyses. 

Crew  Station  Panel 

•  Traceable  crew  Pull  data  are  available  for  fragments 
and  projectiles  (toxic  gases,  overpressure,  and  bums  not 
considered). 

•  JLF  crew  station  tests  will  provide  test  data  support. 

•  Ejection  seat  pyrotechnic  P^,  data  are  not  available. 

•  Cockpit  controls  and  displays  P^  data  are  not 
available. 

•  Liquid  oxygen  (LOX)  converter  P^  estimates  are  of 
low  confidence. 

•  Windscreen/canopy  P^*  data  developed. 

•  For  the  on-board  oxygen  generating  system  (OBOGS), 
on-board  inert  gas  generating  system  (OBIGGS),  flat  panel 
displays,  there  are  no  P^,  or  test  data. 

Engines  and  Accessories  Panel 

•  P*  data  have  kept  pace  with  new  designs. 

•  Test  data  form  important  Pw  input 

•  Large  analysis  data  base  exists  on  obsolete  and  current 
designs. 

•  Confidence  level  is  high  for  older/current  designs  (ex¬ 
ceptions:  large  engines,  fan  sections,  and  small  unguided- 
aerial-vehicle  engines). 

•  Fuel  ingestion  analytical  technique  is  available. 

Stores,  Ammunition,  and  Flares  Panel 

•  Bombs  and  missile  warheads 

1.  Results  are  design  and  threat  specific, 

2.  Analyses  predict  bum  and  prompt  reaction 
thresholds. 

3.  Intermediate  zone  requires  added  tests. 

•  Propellant  and  rocket  motors 


1.  Results  are  design  and  threat  specific. 

2.  Analyses  predict  bum  and  prompt  reaction 
thresholds. 

3.  Complex  propellants  require  added  tests. 

•  Bares,  ammunition  drums,  and  ammunition  boxes: 
limited  test  data  and  estimates 

Electrical  and  Avionics  Panel 

•  Most  work  (test  and  analysis)  dates  from  1983  or 
before. 

•  Component  design  technologies  date  from  the  1950s 
and  1960s. 

•  A  large  test  base  exists,  but  for  ballistic  resistance 
(VM)  development. 

•  More  than  90%  of  the  tests/analyses  are  for  fragments. 

•  No  P^,  estimates  exist  for  many  common  components. 

•  Techniques  are  adequate  for  penetration  related  kill 
criteria. 

•  Pw,  for  some  lethal  mechanisms  are  only  obtainable 
from  tests. 

•  Avionics  and  electronics  are  flight  critical  for  many 
new  aircraft  designs. 

Structures,  Landing  Gear,  and  Armor  Panel 

•  Structure 

1.  Analysis/data  are  not  easily  mapped  into  vulnera¬ 
bility  analyses. 

2.  Models  are  largely  for  skin  panels  and  simple 
structure. 

•  Landing  gear 

1.  Few  test/analysis  data  exist  (not  flight  critical). 

2.  Some  data  are  available  for  Army  truck  tires. 

•  Armor 

1 .  Data  are  experimentally  derived,  so  confidence  lev¬ 
els  are  high. 

2.  Most  data  are  for  projectiles  and  current  materials. 

Helicopter  Unique  Components  Panel 

•  Analysis  techniques  are  valid  for  metal  rotor  blades. 

•  Component  Pw  data  are  mostly  for  fragments  and 
armor-piercing  projectiles. 

•  Flight  controls  can  use  aircraft  techniques  and  data. 

•  Tail  boom  structural  analysis  capability  is  poor. 

•  Little  recent  test/analysis  has  been  done  for  Pw  devel¬ 
opment 

•  Extensive  tests  are  needed  for  failure  at  specified  con¬ 
ditions. 


ACRONYMS 


AAA 

anti-aircraft  artillery 

AFV 

armored  fighting  vehicle 

AP 

armor-piercing  projectile 

API 

armor-piercing  projectile  with  incendiaries 

BAST 

Board  on  Army  Science  and  Technology 

BRL 

Ballistic  Research  Laboratory 

BRL-CAD 

Ballistic  Research  Laboratory  Computer-Aided  Design  program 

COVART 

Computation  of  Vulnerable  Area  and  Repair  Time 

DDDR&E 

Deputy  Director,  Defense  Research  and  Engineering  (Test  and 

(T&E)/TWP 

Evaluation)/Tactical  Warfare  Programs 

DEM/VAL 

demonstration  and  validation 

DoD 

Department  of  Defense 

DoDI 

Department  of  Defense  Instruction 

DT 

developmental  tests 

DT&E 

development  test  and  evaluation 

EMD 

engineering,  manufacturing  and  development 

FEBA 

Forward  Edge  of  the  Battle  Area 

FMEA 

Failure  Mode  and  Effects  Analysis 

FTA 

Fault  Tree  Analysis 

FY 

fiscal  year 

GAO 

U.S.  General  Accounting  Office 

HE 

high  explosive 

HEI 

high  explosive  with  incendiaries 

HEIVAM 

HEI  Vulnerability  Assessment  Model 

HEVART 

High  Explosive  Vulnerable  Area  and  Repair  Time 

IDA 

Institute  for  Defense  Analyses 
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ACRONYMS 


IOT&E 

initial  operational  test  and  evaluation 

JLF 

Joint  Live  Fire 

JSEM 

Joint  Services  Endgame  Model 

JTCG/AS 

Joint  Technical  Coordinating  Group  on  Aircraft  Survivability 

JTCG/ME 

Joint  Technical  Coordinating  Group  for  Munitions  Effectiveness 

LFT  ' 

Live  Fire  Test 

LFT&E 

Live  Fire  Test  and  Evaluation 

LOX 

liquid  oxygen 

LRJP 

low-rate  initial  production 

MECA 

Modular  Endgame  Computer  Assessment 

N/A 

not  applicable 

NRC 

National  Research  Council 

OBIGGS 

on-board  inert  gas-generating  system 

OBOGS 

on-board  oxygen-generating  system 

ORD 

Operational  Requirements  Document 

OSD 

Office  of  the  Secretary  of  Defense 

PM 

Program  Manager 

SAM 

surface-to-air-missile 

SQuASH 

Stochastic  Qualitative  Analysis  of  System  Hierarchies 

STAR 

System  Threat  Assessment  Report 

SURVIAC 

Survivability/Vulnerability  Information  and  Analysis  Center 

T&E 

test  and  evaluation 

TBD 

to  be  determined 

TEC 

Test  and  Evaluation  Committee 

TEMP 

Test  and  Evaluation  Master  Plan 

WPAFB 

Wright-Patterson  Air  Force  Base 

wwn 

World  War  II 
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